Future of this project

[ahuggins-nhs]

This EDI X12 library is going to enter maintenance mode. This is for multiple reasons which I'll lay out below. The good news: this will remain in maintenance mode until either js-edi hits a stable version 1.0.0 (more on that later) or the community wishes to carry this library forward.

The reasons I'm not happy with the current state of this library (which is STILL great, thanks TrueCommerce and @DotJoshJohnson, this is still the superior JavaScript library for ASC X12):

1. The parser in this library depends on utf8, despite some effort to correctly support ascii properly; this can lead to some unusual behavior with "binary" elements
2. The parser is built around version 4010 with no support for component values or repeated elements
3. Streaming (that I wrote) depends on extending Node Readable; web standards are moving forward from this
4. The JSON EDI notation (that I added) is missing some key features and is separate from the object model
5. The element selector language is based on slow regular expressions and has some features (again, that I added!) which I no longer think should be part of the implementation

It's not that any of this is unmaintainable or wrong (well maybe in a few cases). There just are optimizations that I would like to make and features I think should be a part of handling or parsing EDI that are difficult to add with the current state of the code. Just a few things I want:

1. Support for X12 version 5010 and up
2. Support for EDIFACT (! and which is outside the scope of this X12 lib)
3. Faster and more efficient querying for elements (specifically to speed up transforming to JS objects)
4. Support for binary elements
5. Flexibility with emitting different objects and events during parsing and generating
6. A fully-realized and consistent document object model
7. Flexible validation using JSON Schema instead of the current custom format or XML Schema (god please no XML)

Given all of this, I started on an experimental parser written in ANTLR4 grammar. Once I had the parser fully functional, I added an (unfinished) EDIFACT parser in ANTLR4 grammar as well. Then, I reimplemented an object model, and rewrote the element selector language in ANTLR4 and...

Now I'm getting somewhere with the project that I'm calling js-edi. The parser appears to be faster, supports more features of EDI X12, and seems very flexible. The element selector implementation (in progress) is blazing fast. Everything is in a state of flux (especially documentation) since the library is immature, but I'm at a point where I would really appreciate community feedback and community contributions, as well.

• What features are you using daily in this library that should make it to js-edi?
• What features would you want to see in a "universal" EDI format handler?
• What would you like to see for the future of node-x12?
  • Should this library live on as-is
  • Adopt the x12 parser
  • Be retired when the new library is mature
  • Other

It's been awesome to have been able to maintain, enhance, and add features for the last 2 years with the community for this library. Thanks again to @fpw23 @DotJoshJohnson @boxfoot and other contributors and users. I look forward to discussion on this issue.

[fpw23]

I think there is a need for a solid EDI Parser/Generator in JS. node-x12 is by far the best out there and I for one am very appreciative you kept it going but it's got some dust on it just due to being old. I think you have made a great case for creating a totally new project instead of making major changes to this one. My opinion would be to just retire this project once js-edi is stable.

I can't really contribute to requested features yet because my app using node-x12 is still in development (🤞 we go live in a few months). Once we are live I will be more than happy to give better feedback.

[boxfoot]

@ahuggins-nhs Thanks for letting us know. This all makes total sense, and I'm excited to see js-edi take shape (will start following the repo!). I really appreciate your work maintaining and expanding this library up until now and your investment in open tooling for X12/EDI in general. Please let me know (through issues or discussion on the js-edi repo?) if there are particular contributions that would be helpful at this point.

[boxfoot]

Regarding your questions -- for our use, I ended up writing a layer around node-x12 that provides named access to elements in the EDI message. Basically, you can do things like this (pseudoCode):

const transaction = parseTransactionAs278(rawMessage);
const patientFirstName = transaction.subscriber.NM1.firstName;

It's written with Proxies, so while it looks like dot-notation, behind the scenes it's just JSEDI notation with smart getters (and setters, for updating/building the message -- but I only have a couple of those in place). The current implementation is crufty, especially when it comes to components (within elements) and hard-coded to the transactions that we typically use, but I wonder whether a tool like this might be helpful as part of JS-EDI -- maybe not core but as part of the suite. Since we can't distribute the schemas with OSS, ideally we'd be able to offer a CLI to convert the schemas from X12/WPC into a format that could be loaded by the library. I'm curious if others would find something like this helpful.

The part of node-x12 that I depend on the most is serialization of an X12 Object / JSEDI into a valid message with all of the delimiters, control numbers, and segment counts in place, and would hope that similar functionality makes it into js-edi.

[DotJoshJohnson]

Great work, @ahuggins-nhs! I never really anticipated node-x12 being useful outside of TrueCommerce as it was hastily written to support an internal VS Code extension. I'm happy to see it was given a second life! Best of luck with js-edi!

[ahuggins-nhs]

Regarding your questions -- for our use, I ended up writing a layer around node-x12 that provides named access to elements in the EDI message. Basically, you can do things like this (pseudoCode):

const transaction = parseTransactionAs278(rawMessage);
const patientFirstName = transaction.subscriber.NM1.firstName;

It's written with Proxies, so while it looks like dot-notation, behind the scenes it's just JSEDI notation with smart getters (and setters, for updating/building the message -- but I only have a couple of those in place). The current implementation is crufty, especially when it comes to components (within elements) and hard-coded to the transactions that we typically use, but I wonder whether a tool like this might be helpful as part of JS-EDI -- maybe not core but as part of the suite. Since we can't distribute the schemas with OSS, ideally we'd be able to offer a CLI to convert the schemas from X12/WPC into a format that could be loaded by the library. I'm curious if others would find something like this helpful.

The part of node-x12 that I depend on the most is serialization of an X12 Object / JSEDI into a valid message with all of the delimiters, control numbers, and segment counts in place, and would hope that similar functionality makes it into js-edi.

@boxfoot I like the idea of providing similar functionality to the dot-notation/named property style. I've used Proxy class similarly in an unrelated project https://github.com/aaronhuggins/kodi-api/blob/main/lib/KodiClient.ts. Given the validator that I recently finished (I think it's finished) that can validate an EDI document that has been parsed using JSON Schema, I think we could probably tie that with a given JSON Schema or interface that the proxy could use to access the underlying DOM. What would be REALLY powerful, is if we could provide a toJSON method so that the given class instance also serialized to JSON as the end user would expect.

[ahuggins-nhs]

I can't really contribute to requested features yet because my app using node-x12 is still in development (🤞 we go live in a few months). Once we are live I will be more than happy to give better feedback.

Is it worth it to have some level of compatibility with this library to make transition to js-edi any easier? Or just a clean break?

[fpw23]

I think a clean break would be better, in my project I abstracted the use of node-x12 away enough to easily support changing it out.

[pomSense]

@ahuggins-nhs Thanks for your contributions to this project and very excited to see the transition to js-edi. We went live in production with this few weeks back and so far it is very solid. Our use-case is primarily dealing with parsing EDI214s and normalizing the data to our application data model. I've dived in the world of EDI recently so take the feedback from that perspective:

• What features are you using daily in this library that should make it to js-edi?
  • X12Parser -> getInterchangeFromSegments
  • Transform streams
• What features would you want to see in a "universal" EDI format handler?
  • I think a flexible parser that parses EDI directly to JSON based on the fields present. Would make developer UX seemless
  • EDI validator, parser
• What would you like to see for the future of node-x12?
  • Honestly, I think node-x12 is pretty good. I think the only thing that would next-level this is the same feature I mentioned where the parser is flexible enough to auto-generate a JS object based on transactions, segments and loops. Here is an example of how our 214 map looks like. Its a basic map, but gets redundant to handle a wide breadth of EDI formats that different clients are sending:

export const EDIx12Map: IEDIX12Map = {
  ST: {
    // Transaction Set Header
    edi_type_code: 'ST01',
    edi_control_number: 'ST02'
  },
  B2: {
    // Beginning Segment for Shipment Information Transaction
    traffic_service_code: 'B201',
    carrier_scac: 'B202',
    standard_point_location_code: 'B203',
    shipment_id: 'B204',
    weight_unit_code: 'B205',
    payment_terms_code: 'B206'
  },
  B2A: {
    // Set purposes
    transaction_set_purpose_code: 'B2A01',
    application_code: 'B2A02'
  },
  NTE: {
    //  Note/Special Instructions
    note_reference_code: 'NTE01',
    free_form_description: 'NTE02'
  },
  B10: {
    // Beginning Segment for Transportation Carrier Shipment Status Message
    carrier_load_id: 'B1001', // Carrier PRO # or invoice number
    client_load_id: 'B1002', // the shipper's master BOL number
    carrier_scac: 'B1003', // : Standard Carrier Alpha Code (SCAC)
    inquiry_request_number: 'B1004',
    reference_id_qualifier: 'B1005',
    reference_id: 'B1006',
    response_condition: 'B1007'
  },
  L11_72: {
    // Business Instructions and Reference Number
    reference_id: 'L1101:L1102["72"]', //L11 has qualifiers
    reference_id_qualifer: 'L1102:L1102["72"]',
    description: 'L1103:L1102["72"]'
  },
  L11_QN: {
    reference_id: 'L1101:L1102["QN"]', //L11 has qualifiers
    reference_id_qualifer: 'L1102:L1102["QN"]',
    description: 'L1103:L1102["QN"]'
  },
  L11_BM: {
    reference_id: 'L1101:L1102["BM"]', //L11 has qualifiers
    reference_id_qualifer: 'L1102:L1102["BM"]',
    description: 'L1103:L1102["BM"]'
  },

...
///extends another 150 lines
...

SPO: {
    //Shipment Purchase Order Detail
    purchase_order_number: 'SPO01',
    reference_id: 'SPO02',
    unit_measurement_code: 'SPO03',
    quanity: 'SPO04',
    weight_unit_code: 'SPO05',
    weight: 'SPO06',
    application_error_condition_code: 'SPO07'
  },
  SE: {
    // Transaction Set Trailer
    number_of_included_segments: 'SE01',
    set_control_number: 'SE02'
  }
}

• Should this library live on as-is
  • I hope it does! We just went to production with it. At least maintenance mode at worst! I am happy to help contribute once I level up my coding abilities.

I'm happy to provide more feedback if needed. But thanks so much for this library, you don't realize how much this has helped us out.

[dejavxtrem]

@ahuggins-nhs this is awesome. I am an EDI developer and a full stack engineer . i would love to contribute the best way i can.

[aaronhuggins]

Hey all, I'm going to be giving this some maintenance over the next month. Among the things I'll be doing:

• Porting this codebase to target Deno, since no X12 library exists yet for that platform and js-edi is not mature
• Cleaning up the codebase and documentation
• Adding in a build script for Deno to Node Tool
• Using dnt to build for Node provides support for both ES Modules and CommonJS
• Adding CI workflows to improve releasing and versioning

Overall, this change should make it easier for me to maintain this library going forward, so it's less time doing "npm install... dammit, vulnerability, why... oh, it's a Node quirk..." and more time doing "Blissfully making my code run correctly on either Node or Deno".

See #37

[aaronhuggins]

Also, if anyone knows anyone who's hiring, I'm open to work. Especially if folks want to hire me to work on EDI parsing/handling.

[dejavxtrem]

@aaronhuggins how are you doing? I am thinking we could collaborate on a project which could be an MVP. Let me know if you are interested. I would love to work with you.

[aaronhuggins]

Hey @dejavxtrem. I was not doing great a few months ago, very sick for quite a while and only recently better. Trying to find work again and being picky about staying primarily Node/Deno focused. My DMs are open to mutuals on Twitter, so if you @ aaronhugginsdev there we can talk.

[dejavxtrem]

@aaronhuggins i am not on twitter. but we can connect via email: dejavxtrem @ hotmail.com . Send me an email when you can