Phase two round two research findings
-
Explore ways to allow users to find guidance by activity or scenario, regardless of document type.
-
Explore and test options for showing documents that are outdated. Users should be able to tell if a document is no longer active before opening it.
- Prioritize clarifying the effective date in the UI for current and previous versions of regulations.
- Navigation largely works for external users – deprioritize major navigation changes until we can test them.
- Five 45-minute interviews with external users
- Same focus as with internal users – related documents, versioning, and print/PDF
- Users were all compliance managers or consulted with companies about compliance.
- Everyone had at least basic digital literacy, a few were digital experts.
- What type of documentation is most important to external users? Is this different than what internal users reported?
- Do users want to see rulings that have been superseded by newer rulings?
- When do users want to print regulations or parts of regulations? Why?
- When do users want to create PDFs of regulations or parts of regulations? Why?
There is a disconnect between how users want to find information, and how ATF currently presents it. Users want to find information that relates to an activity or scenario.
Until now, ATF has organized its content by document type, which means that when a user comes to the ATF website and wonders "Can I sell ammo to a person from California if I'm in another state?" they don't know which category in the navigation will answer their question. They don't know if it's under Rulings, Open letters, the Rules and Regulations Library, or another section.
"You almost have to know [a document] exists in advance and many people don’t know that."
Even when a user chooses a document type, let's say Rulings, and navigates to the page, they have to wade through a long list of documents without knowing which ones are current, and the still don't know if the information they're looking for is located within any of the other documents.
Ultimately, external users don’t care what document type information comes in. They just want to find the answer to their question.
“We look at the law (the statute) the regulations, the rulings, and newsletter, guides, info on ATF website in html text, handbooks (like from NSA) – any form of medium that contains rule or interpretation.”
Users find newsletters extremely useful but it’s difficult to find past articles.
“I had to call ATF, I know them pretty well, but I couldn’t find an article about steel floor storage. It was in their newsletter about shared storage. I could not for the life of me find it. Maybe not in a newsletter but in an open letter. Unless you get that newsletter there is no way to know.”
“There was a change to a form and it was buried in a newsletter within an email. I would want that on the front page.”
Users were very excited about the rulings in the sidebar.
“I would say from a UI perspective it’s very nice, very easy to navigate.”
“It puts an impetus on the licensees [to be compliant], but if you have been doing your research you should see it.”
And validated the decision to add additional kinds of documents.
“What is missing [from the right-hand column] is the open letter. It would be nice if you could do that as well. It would allow me to know and reference to it with the rulings that relate.”
People disagree on whether they want to see outdated rulings.
“Most of us just want to know what is current.”
“But if you are inspecting me for something I did a year and a half ago I need to know about the old regulation.”
Either way, they want to see a ruling’s status quickly.
[Are you interested in rulings that are no longer applicable?] “Yes, but the presentation of such data should be changed. I would go the site and see this long list of rulings [and not know which ones are superseded].”
“As far as I know [the ruling is] effective. I haven’t been notified that it is not. [There's] no expiration date, nothing that tells me that it’s expired.”
People choose different formats based on task and location. Task falls into three main categories – find, read, and share. (Understand is another main task type, but doesn't seem to influence format preference much).
For searching, users prefer first HTML, second PDF (only if the text is machine-readable), and only use print as a last resort.
“I wish we had a better search engine for info on regulations.”
“The newsletter is not searchable, it’s in a PDF format which makes it hard.”
“I almost always use an online or PDF. I can search more freely.”
For reading, users seem to prefer whatever format is easiest to read.
“The print is bigger [than eCFR] and easier to read.”
People use a variety of techniques for sharing regulation text. Most frequently, people will use a website or PDF version to copy and paste text into an email for sharing. Somewhat less frequently they will send someone a link to a website. Sometimes they will highlight text within a PDF and attach that to an email. People will share information via printed copies, but only in person.
Users prefer the website or a PDF while at their desk and connected to the internet. When away from their desk but still connected (say, in a meeting), they will either pull up a regulation or document on the screen (either website or PDF) or use a print copy. Offline, users will either use a PDF that they have previously downloaded or a print version.
“Rarely [use printed documents]. It’s so much faster and easier to pull it up on the screen.”
“[I] use paper as well if in field, but typically use online.”
“Yes, I do [show my trainees the printed copy] but I also give them the online versions so they can look up the regulations and what not.”
- We saw more confusion around effective date. “I would spell it [the effective date] out for the folks. If I have to explain it, I would say ‘this is the regulation at that time, it may have been superseded but go here and find out.’”
- People had a very easy time navigating the eRegs site. “I like this better [than eCFR] b/c it takes me right to the sections that I’m looking for.”
- External users had about the same difficulty creating a print or PDF from the eRegs site as internal users, although it's important to note that everyone we talked to was reasonably familiar with using a computer.