-
Notifications
You must be signed in to change notification settings - Fork 20
Phase two, round one user research findings
August 2017
Through our research we were able to identify top priorities for users, as well as functions that users aren’t interested in. Top priorities are expressed below as design opportunities, in the form of questions that don’t prescribe any particular solution.
Since we spoke to internal users, we recommend conducting similar research with external ATF users to learn more about how their needs differ.
How might we …
- Help regulations lawyers and legal scholars find rulings (both active and otherwise) and related legal documents, without getting in the way of licensees and investigators in the field who just want to know what they have to do now?
- These users also want to see related statutes, open letters, and internal decisions.
- Help users easily find forms that are mentioned in regulations?
- Allow users, some of whom may be less technically savvy, to print an entire regulation?
- This solution will need to be easier for users than purchasing a copy from the Government Printing Office.
- Help licensees without a law degree understand regulations and rulings so that they can remain compliant?
- Better support industry operations investigators (IOIs) while they are working in the field?
- They often don’t have an internet connection.
- They refer to regulations to refresh their memory.
- The inspector goes over regulations with the licensee (they currently carry a printed version of the regulation).
- They have a closing meeting with the licensee, when they review violations, regulations, and rulings, and they also share supporting materials.
- Support multiple search strategies, including searching by topic or keyword as well as by citation?
Internal users don’t want to print or create a PDF of an individual section or paragraph of a regulation. While it’s true that some users had difficulty understanding how to create a PDF or print a copy of one page of eRegs, they also expressed no desire to print or create PDFs of individual pages from eRegs. Instead they’re only interested in having a print or PDF copy of an entire regulation part or related document (for example, a ruling or open letter). When users want only a snippet of a regulation or document, it’s when they want to copy/paste the text into an email or other correspondence.
Based on the priorities that ATF identified, the team came up with the following framing questions to answer with this round of research –
- What type of documentation is most important to users?
- What is the right amount of information (related documents) to display in the right hand column?
- Do users want to see rulings that have been superseded by newer rulings?
- Print and PDF
- When do users want to print regulations or parts of regulations? Why?
- Do they want to download a file for offline use?
- Are they printing them because they can’t do what they need to do on the web? How do users currently print parts of the regulations?
- When do users want to create PDFs of regulations or parts of regulations? Why?
- How do users currently create PDFs?
We conducted five remote interviews with internal, ATF users who work with policy and regulations in a variety of ways. Each interview was about 45 minutes long.
Field investigators only want to see rulings that are currently active rulings when in the field so they can more easily educate licensees.
“In the field [we only look at rulings that are] active.”
Those responsible for ATF policy also want to see rulings that have been superseded so they can better understand the history of ATF’s position. They also want to see previous versions of regulations and other related documents.
“We always keep the old ones [regulation guides], and sometimes people want to know what the language was in the previous version back to 1989… I think it was updated in ‘85, ‘89, ‘95, ‘01, something like that. Occasionally, we look at the historical guides.”
“I am kind of like the librarian here. I have old copies of the previous regulation guides in our library. Sometimes the question comes up in a rewrite.”
“I have looked up previous versions of regulations, looking and going into federal rights register notices and looking up what’s changed, what we’re changing it to. I forget what project on that was, but if I could go to eRegs and see that, it would cut down so much time because the Federal Register, it’s so many steps to get there.”
“[Seeing the history] helps me understand it more. I took a course on interpreting law, and seeing how it changed helps me understand congressional intent.”
“[When not in the field] We look back [at superceded rulings] to understand the why. […] helps understand it for rulings and inspections.”
Internal users would like to see forms so they can easily refer licensees to the forms they need to use, as well as related plain-language guidance.
“Hypothetically, I want to go to a specific ruling 478.126a with the regulation, the form, the link to guidance around sales. How do these links help people comply.”
“I’d also like to see the forms related to the topic they’re talking about. The forms for industry folks to get those forms easily.”
“One thing to consider would be to, if I am looking at that definition, to possibly link to the relative ATF.gov information? I know that is huge but … you talk about knowing the relevance and knowing where to go to, if I am John Smith out there, if you could link under those definitions, it would be more user friendly for somebody who’s not used to using it.”
“As you get in some specific regs you may want to link up specific forms. Also link up FFL newsletters for related regs.”
Internal ATF users and legal scholars want to look at the original statute to better understand the intent behind a regulation.
“I start with the statute, then regs, then look at internal decisions, sometimes use Q&As for public language.”
“It would also be really nice if the statute is also linked to it. Sometimes regulations are specific to a statute. The statute says whatever, and the regulation will mimic that language—or some of them say things like other records of the attorney general will deem appropriate. The regulation helps fill that out, so it would be helpful to know what it relates back to.”
Users also mentioned Q&As, open letters, procedures, and contact information as information that could be useful.
“It would also be helpful to have the public Q&As.”
“To figure that out [a question about a ruling], I would call the office of regulatory affairs and ask them what it means. I don’t know how the public would find that out, other than emailing us.”
“The other thing would be the contact for who covers this regulation. Who or what generic office should somebody contact if they had further questions.”
“We also put out things called open letters—clarifications on certain topics. They often go back to regulations. If those were there, that would be great.”
“[If you had a magic wand what would you change?] Add reference to applicable docs in the righthand side. ATF procedure and rulings. Ruling a blanket variance or clarifiers what the law provides. Procedures is step by step on how to comply.”
“[Any other docs you want to see?] Supplementation docs the the closing conversation. Best practice guide, safety security packet, contact info, outlines, fulfill sales.”
Users generally found the interface with the right-hand column to be pretty easy to use and understand.
[Had issues getting back to ruling with the multiple tabs.]
“I would want to look at the ruling to figure out what it deals with, it’s pretty easy here because I can read the regulation on eRegs and see what it talks about.”
“Rulings are ordered most recent first. Order makes sense and is help but there should be more rulings.” [currently rulings are displayed in reverse chronological order]
“I would put them [rulings] in chronological order. “
“They [the rulings] are in reverse chronological order.”
“When you click on a particular section, it populates any ruling that deals with that section.”
“I would like to see the ruling it amplifies, somewhere you could access it or reference it if you weren’t familiar with it.”
Internal users only want print versions of an entire CFR part – they aren’t interested in printing out individual CFR sections from eRegs.
When they do refer to a few paragraphs of a regulation, it’s generally in digital form so they can send a link or copy/paste the text into a letter or email.
“I wouldn’t [print] normally a singular reg. Unless there was a specific question and I need to provide a write doc around a specific reg and then can copy and paste from a single reg.” [in this example they used ‘reg’ to mean a single CFR section in the eRegs interface]
“It would be a real pain to have to print each one of these regulations separately. For the field, I would want a way to print all the regulations without having to print them separately.”
“I would only need to print it so I can see the really long sections and have it all in front of me at one time, if I really wanted to analyze it.”
“We wouldn’t really print out a page of a particular regulation.”
Internal users print CFR parts (and other documents) to send to licensees or the public, or to analyze or edit long passages. In some cases, internal users already have a hard copy of the part.
“And [we] used to give hard copy but now advise folks to get hard copy at GPO ($35) or download it off our website and take it to Kinkos. […] Many folks want hard copies. We try to provide options.”
“As we move forward, more and more people have iPhones and iPads and electronic records. But we also have old school FFLs [federal firearms licensees] that existed before the Gun Control Act. They may be old school and very grounded in paper.”
“Technically, we print those [rulings] out when we’re sending out letters for correspondence. The actual regulation, not that often. They get mailed out to licensees or the public.”
“Or if I wanted to edit it or amend it, I would definitely print it out.”
“Printing: not very often at all. We do periodically get the paperback CFRs. ATF puts out the reference guide on the firearms side. Federal explosives laws and regulations on explosives side. It’s hard to get paper copy, but I am old school and like my old books. You can download them from the website. I’m usually pulling up a book or looking at the website, not printing them out.”
“I would probably have the print version, what we call the orange book, in hardcopy in my hand and downloaded prior to not having Wi-Fi.”
Some internal users know how to use a browser’s innate print function to print from eRegs, others add unnecessary steps, and still others are unable to figure out how to print at all.
“I don’t see a link to print. Let’s see what I get when I right-click it and look at it in the print format… Hmm, it doesn’t give me a preview so I don’t know what it’s like when I print it out.”
“I don’t see any print option, so I would copy and paste it into Word, and then print it.”
“To print, I would go to GPO [Government Publication Office]. I’ve never tried to print off of eRegs.”
“[How would you print?] Copy and paste or used control+P.”
“With Windows, when you’re on a webpage, I would use the little gear and then choose Print.”
“To print, I would right-click > Print Preview.”
Internal users use PDFs to attach to emails or to keep a regulation handy for easy retrieval. Sometimes they will create a PDF and sometimes they will download the ‘official’ PDF.
“I never create a PDF. Usually, if I wanted one, I would find the official version online.”
“If I was using it regularly, I would go to copy and paste or make the PDF.”
“I do a lot of research on topics, and I have a file system on my computer. If I was trying to save something as a PDF that I might not be able to find it again, I would put it into a folder so I can find it easily. I occasionally save something like this.”
“But for information collection, if I could just save or print or convert the regulation to PDF, I would use that. Now, I have to [paste] it into OneNote or a Word doc and create the PDF from there.”
“I have absolutely no idea how to print a PDF of this.”
“[PDF?] Ctrl+P then save as PDF.”
Users think a lot about whether or not the source information is trustworthy, but they have different ways of assessing that.
“One concern that we have—and this is lawyers being nervous nellys—every time you recreate it, you risk creating an error. One of the hesitations is we have this all out there. Why are we recreating it again where we could make an error? It’s one of my hesitations. If you’re going to have one place, let’s do it.”
“I stick to Lexis or an agency’s website, because sometimes you don’t know [if you’re looking at the right version]. I stick to websites I know are the authorities.”
“I felt confident that it was up to date, because I know that info gets pulled from the official online regs, which aren’t the official ones on paper, but when the online ones are updated, eRegs is updated.”
Internal ATF users rely on their day-to-day familiarity with regulations and related documents to understand them, and to know if the information they’re seeing is current, complete, and accurate.
“I knew that that section hadn’t changed recently. It’s really just trade knowledge. I work closely with the Office of Regulatory Affairs, and ATF puts out a publication/reference guide, and I’ve been tasked with updating that in 2014 and 2017.”
“I knew it was up to date, mainly because I work with them so much that I know when changes are made. […] There haven’t been a lot of changes, and I am aware of that going in.”
“I felt confident that it was up to date, because I know that info gets pulled from the official online regs, which aren’t the official ones on paper, but when the online ones are updated, eRegs is updated.”
“Basically it’s [knowing a reg is up to date] staying in tune with notice and announcements and cross referencing with online. I get the notices via email but they are online.”
Internal ATF users generally look up regulations when they are responding to questions or inquiries (via email, letter, or other channel), or when they are doing investigations in the field.
Often they are familiar with the regulation in general terms but want to refresh their memory or copy the exact language.
“There may be questions about “what does this mean” or “can I do X or Y under this ruling”—usually because they can’t.”
“Not learning so much as reminding myself of the words. Reviewing the language to make sure the determination we’re coming from aligns with the language from Congress in the statute and the regulation itself.”
“When I worked in the first as an industry operations investigator, I was out doing compliance work. You don’t always have internet or cell phone access. They can be rural or places where there’s no electricity or no heat. In the field, it’s really important to have that copy available to look things up and answer questions.”
Internal users benefit from that familiarity when navigating the eRegs interface – external users may have a more difficult time.
“From experience and repetition, we know where it is. People may tab them or just know where it is.”
“Sometimes people fail to read to the end of the ruling, and they’ll be trying to apply it to something that’s excluded, like a particular group of firearms even though the ruling specifically says it excludes them.”
“I know they’re relevant, we try to address that in the title by making it clear when we collectively write these. Sometimes it’s a matter of clicking on it and seeing if it’s relevant to what you’re working on.”
“Rulings are the agency rulings. I haven’t seen any of these. This is really cool. I don’t know how they relate. I don’t know why these are matched with this [regulation].”
“I almost always have tangential questions, but I never feel like I know everything.”
“A co-worker had asked for information by email, I was trying to find a specific regulation number to provide it to her. I knew kind of where it was but didn’t know the exact citation. I scrolled through the table of contents and then provided her with that information.”
“I didn’t look at any of the original text. I was looking for the cite number, regulation number so she could read it for herself.”
Other times users will search by keyword or topic – this seems to be more true the less familiar with a regulation a person is.
“I may not be sure where to look, so I’ll look by a particular issue rather than a specific regulation.”
“I usually use Control-F and look for specific words.”
“If I don’t know what I’m looking for, I pull it up online and use the search tool to look for a particular keyword. It’s in a PDF version on our website.”
“I would like to see the all of the rulings at the same time. This seems too confined, and confines me, only one section at a time. If I’m doing research on a certain issue and I’m not familiar with all of the sections, I would rather Control-F and search through it [to find what I need].”
“We mark off sections we refer to frequently. It’s a matter of copying it and pasting it, or retyping it from paper etc. Not looking at them digitally.”
“We mark off sections we refer to frequently. It’s a matter of copying it and pasting it, or retyping it from paper etc. Not looking at them digitally.”
“ATF publication, Federal Firearms Regulations Guide. It’s online at atf.gov, and it contains rules and regulations around firearms. There are times when I keep a hard copy at my desk, and we all do that work here that do what I do.”
ATF tends to get more questions when a ruling first gets released, or around topics where technology has outpaced the regulation.
“When a ruling first gets released, we do tend to get questions about them because a lot of it’s new.”
“I can remember naively joining ATF in 2006, firearms have been around a long time, how many questions can there be? It’s not so much about the regulations, as what we’ve seen before versus new issues. You have to take all of those tools and make an interpretation or determination if it’s a new issue.”
“Question [about] auto populating Form 4473. Background check. The question came from an internal policy discussion from a tech policy group. Basically the tech firm inquired about licensed dealers, can they scan drivers license and enter certain fields into forms. They were hired by industry members. I was away from my desk and had the book and looked it up.”
ATF staff refer to regulations and related documentations when responding to inquiries – they copy/paste short texts from the regulation into their letter and sometimes include links or attachments.
These responses can be either via email or paper letters.
“If we did move to email, it would be nice to have the link to the letter and the regulation, so they could go and pull that.”
[two types of inquiries about regs:] “Congressional inquiry comes through a congressman from a constituent. General inquiry come come from anyone in the public, students, prisoners, etc …”
“I might look up a regulation to reference it when we’re writing about something in correspondence. I might just look it up. For example, making sure either I was understanding the regulation correctly or copying the regulation to insert it into a letter.
“Copy and paste will likely go into the congressional process. General inquiries will go back the way they came.”
Field investigators go over regulations and related documents with licensees during inspections – they usually leave behind paper copies.
“Field IOIs look them up on a daily basis and may have to look up something on the spot. 99% can speak in general terms. Depending on the industry they will need to ref the book. I had to know regs inside and out with the nature of my job.”
“By law we do inspecting for explosives every 3 years. Initial inspection with firearm around qualifications around zoning, age, inventory. Then interview around proposed business and go over acknowledgement of regs and once they get their license inspection is gauged around a variety of scenarios.”
“During a closing conference they will describe in detail violation, how, often they occurred, and will refer back to the reg. They are somewhat bombarded with the “acknowledgement” and going through material on that form IOIs will handouts, copies of guidelines, school zone safety acts, rulings, newsletter, safety and security book.”
“Me it’s [referring to regulations in job] daily application. {Hard to hear} Educate arms industry. Daily basis guide and educate folks and have access to the regs on daily basis. Education element ‘required to XYZ’”
“Whether it’s the industry or general public, we hear that the regulations are hard to understand because they sound like lawyers wrote them because lawyers did write them. They’re not in plain language, so it’s in legalese.”
“After two years in law school, hopefully I can understand regulations.”
“Status of ruling: it hasn’t been updated or rescinded. This is the most current.”
“Status of ruling: I’ve never seen “active” described. I’m guessing that means it’s authoritative or final, but I actually don’t know.”
“I don’t know what “amplified” meant, it must be specific jargon to ATF. Maybe it means it’s expanded by that other ruling.”
“1977-1: I’ve never seen the term “amplified.” We usually have “superseded,” and I have absolutely no idea what that means.”
Internal users understand the numbering system for rulings but this is not necessarily true for external users.
“Our rulings are numbered by years, so then we give it a number after based on the number of rulings that were made that year. Newest to oldest order on the page.”
“I’m not familiar with ATF rulings. I’m guessing they’re chronological. I’m not sure what these [appended] numbers are, I’m guessing it’s a citation to the ruling.”
These are observations that came up but not frequently enough to draw any real conclusions. We should pay attention to see if they come up again in future research.
“Effective date: this is kind of misleading. It changes it all to 2003.”
“What’s the significance of the effective date on the site?”
“[If you had a magic wand, what would you change?] For me, it looks like the rulings are only the rulings that apply to the section of the regulation that’s highlighted. Maybe there’s a separate place on our site to get ALL of the rulings.”
“I usually start with Lexis and look at the statute, and then I’ll try to find the regulations. They’re usually harder to find, to see who has authority like what agency, etc.”
“It’s a little difficult to find regs over statutes—those are more accessible and easier to find on Google.”
ATF eRegulations is part of a larger platform. Find out more at https://github.com/18F/eregs-platform/wiki