Define and extend N1MM communication protocol with N2IC

[w7sst]

Summary

Extend the N1MM-MorseRunner protocol to support additional contests (e.g. Arrl Field Day).
This will build upon N2IC's work to provide an integration of Morse Runner into N1MM Logger.

Motivation

Adding multi-contest support to N1MM/Morse Runner provides a valuable training and practice tool to prepare for upcoming contests or for training new and inexperienced contesters.
Members of the N1MM user community have also expressed interest in additional contest support within N1MM and Morse Runner.

Detailed Description

The goal here is to extend N1MM-MorseRunner communications protocol to support multiple contests when running N1MM's Morse Runner integration feature. Ideally, additional contests can be added to MorseRunner without N1MM code changes.

• Work with Steve, N2IC, to extend existing communications protocol between N1MM and Morse Runner.
• Steve and I have been communicating regarding this topic. Steve is ready to receive a code drop from Mike and try to get it running.
• Steve will like make a change or two and pass those back to Mike
• Mike is hoping Steve can provide an early version of N1MM for local testing.
• Review and discuss ideas for generalizing the support to allow contest support within N1MM without N1MM code changes.

Proposed Changes

Below are my proposed changes to existing protocol. In the short term, we will verify that protocol changes can be supported by N1MM and Morse Runner (and perhaps DxLab?). After we verify this approach, we can explore a more generalized approach for the long term. These messages are sent by N1MM and handled within Morse Runner's WinProc handler in Main.pas. A more detailed description of these changes will be sent to Steve, N2IC, once we get started.

Short Term (as part of feasibility study)

1. Add WM_SETMYEXCH (replaces WM_SETMYZONE) - Replace the existing WM_SETMYZONE message with a more generalized approach to include the entire contest-specific message similar to the Exchange specification in N1MM's Contest Setup dialog.
2. Add WM_ARRLFD - Add message to start the ARRL Field Day simulation similar to the existing WM_CQWW and WM_CQWPX messages. Or, directly implement WM_SETCONTEST and WM_RUNPILEUP messages (see Longer Term below).
3. Add WM_SETEXCH1 (replaces WM_SETRST) - message to send exchange field 1 keystroke information (e.g. FD Class, RST, OpName, etc.).
4. Add WM_SETEXCH2 (replaces WM_SETNR) - message to send exchange field 2 keystroke information (e.g. FD Section, Serial number, CQ-Zone, Age, etc.)

Longer Term (as part of final implementation)

5. Add WM_ISCONTESTSUPPORTED - query if a given contest is supported. Returns true or false.
6. Add WM_SETCONTEST - Select a specific contest. Replaces contest-specific messages WM_CQWW, WM_CQWPX and WM_ARRLFD with a generalized message to select a specific named contest. Returns error if not supported.
7. Add WM_RUNPILEUP - Starts the specified contest in Pile-Up mode. Returns error if invalid exchange string is not valid for the selected contest.
8. Add WM_RUNSINGLE - Starts the specified contest in Single-Call mode. Returns error if invalid exchange string is not valid for the selected contest.
9. Define Error Protocol - extend protocol to return error strings back after a Windows message call (e.g. the "set my exchange" message, SETMYEXCH, may fail due to errors in the specified exchange string).

Note that specific changes to existing protocol will be documented in the checkin notes attached to this issue.

Additional context

The following diagram shows the existing protocol between N1MM and Morse Runner. The key startup, QSO processing and shutdown messages are summarized below. The proposed protocol changes are highlighted below in green; existing protocols are highlighted in blue.

Note I learned how to embed a UML diagram into Markdown using Mermaid and couldn't resist the opportunity to try it out. More info here and here.

sequenceDiagram
autonumber
actor User
participant N1MM
participant MR
User->>N1MM: User starts N1MM
activate N1MM
User->>N1MM: User opens Config/Setup Morse Runner dialog
activate N1MM
%%rect rgb(191, 223, 255)
User->>N1MM: User clicks "Start Morse Runner"
deactivate N1MM
N1MM->>MR: Execute Morse Runner
activate MR
  note over N1MM,MR: Startup protocol
  N1MM->>MR: WM_SETMYCALL "N2IC" (set user's call)
  alt existing contest-selection protocol
  rect rgb(191, 223, 255)
    N1MM->>MR: WM_SETMYZONE "3" (set user's exchange information)
    N1MM->>MR: WM_CQWW (starts CQWW contest simulation)
  end
  else *proposed contest-selection protocol change
  rect lightgreen
    N1MM->>MR: WM_SETMYEXCH "3A OR" (set user's exchange information)*
    N1MM->>MR: WM_SETCONTEST "FD" (selects ARRL FD contest)*
    N1MM->>MR: WM_RUNPILEUP (starts ARRL FD contest simulation)*
  end
  end
  activate MR
%% for each QSO...
loop for each QSO
note over N1MM,MR: Typical QSO protocol
User->>N1MM: User clicks F1 to call CQ
N1MM->>MR: WM_KEYDOWN.KeysF1 (request to send CQ)
activate MR
MR-->>-User: audio (CQ is heard by user, DX stations heard calling)
User->>N1MM: User enters call, exchange info, and finally TU
N1MM->>+MR: WM_SETCALL (set received call)
alt existing exchange message protocol
rect rgb(191, 223, 255)
N1MM->>MR: WM_SETRST "599" (send exchange info, RST)
N1MM->>MR: WM_SETNR "05" (send exchange info, NR)
end
else *proposed changes to contest exchange protocol
rect lightgreen
N1MM->>MR: WM_SETEXCH1 "3A" (send exchange field 1, FD class)*
N1MM->>MR: WM_SETEXCH2 "OR" (send exchange field 2, FD section)*
end
end
N1MM->>MR: WM_KEYDOWN.KeysInsert (enter key moves to next phase of QSO)
MR-->>User: audio (user hears subsequent exchange elements)
deactivate MR
end
%% User shutdown sequence...
User->>N1MM: User stops Morse Runner
note over N1MM,MR: Shutdown protocol
N1MM ->> MR: WM_STOP (stop simulation)
deactivate MR
N1MM ->> MR: WM_END (close Morse Runner)
deactivate MR
deactivate N1MM

Loading

Startup protocol

• Message 4 - N1MM executes Morse Runner as a background process.
• Message 5-6 - initialize various contest-specific settings in Morse Runner.
• Message 7 - starts the contest simulation. Morse Runner starts sending audio back to the User.

Proposed Revision to startup protocol

• Message 8-9 - sets user's contest exchange (3A OR) and contest selection (ARRL FD).
• Message 10 - starts contest simulation in Pile-up mode

QSO loop

• Message 12 - send CQ message to Morse Runner
• Message 15 - N1MM sends callsign keystroke information to Morse Runner.
• Message 16,17 - N1MM send exchange field keystroke information (RST, NR) to Morse Runner.
• Message 20 - KeyInsert advances Morse Runner to next step within the QSO. Each step produces new audio.

Proposed Revision to exchange field protocol

• Message 18,19 - N1MM sends exchange fields 1 and 2 keystroke information (3A OR) to Morse Runner.

Shutdown protocol

• Message 23-24 - N1MM sends shutdown messages to Morse Runner.

[w7sst]

Hi Steve,

Fetching these changes into your local repository

I have pushed branch "fd-proto-n1mm-mr" into this repository. I did not create a pull-request into your repo because I don't want this branch to be accidently merged into your mainline.

To get this branch, you can type (from your local repository):

git remote add w7sst-repo https://github.com/w7sst/MorseRunner
git fetch w7sst-repo +fd-proto-n1mm-mr:fd-proto-n1mm-mr
git checkout fd-proto-n1mm-mr

Compile and test

Let me know if you are not able to get this to compile. These changes should run with your version of Lazarus.

To run MR in standalone mode in FD mode, edit your MorseRunner.ini file and change ContestName=cqww to ContestName=arrlfd. This will enable FD mode for you if you want to try it out.

Let me know how your changes with N1MM proceed when you get a chance to work on it this fall/winter (after your busy summer). I am very interested in getting a bootleg copy of N1MM for additional testing.

To push changes back to me

If you have changes to send back to me, there are several choices:

1. zip up your modified files and send them to me.
2. Push your changes to your repository and I can pull them into my local repository.

Steps to push your changes to your repository (I think this will work):

git checkout fd-proto-n1mm-mr
local edits...
git add Main.pas ...
git commit -m "comment regarding changes"
git push origin fd-proto-n1mm-mr

The above assume that your remote is called origin. The check, use git remote -v to find the name of your remote.

[w7sst]

New Windows messages

New Messages:

• WM_SETMYEXCH - set user exchange

    Syntax: WM_COPYDATA WM_SETMYEXCH <sent-exchange>
      where:
        sent-exchange = Exchange value entered in the Morse Runner
                    Setup dialog (currently called "Zone"). Ideally this field
                    is populated with the "Sent Exchange" field entered on the
                    "Contest" tab of N1MM's Contest Selection dialog(s).
                    Like N1MM, the RST is omitted from this string.
                    Examples: '3', '#', '3A OR', 'Mike OR', etc.                                       .

This message replaces WM_SETMYZONE using a contest-specific exchange string.
This message sets the contest exchange value. It is called before the
WM_CQWW, WM_CQWPX OR WM_ARRLFD messages.

Question: How do we return an error string for syntax error in exchange
string?

• WM_ARRLFD - start Arrl Field Day contest

  This message adds support for Arrl Field Day similar to the existing
  WM_CQWW and WM_CQWPX messages. The new WM_SETMYEXCH is used to specify
  the field day exchange. The older-style WM_SETMYZONE should not be used.

Suggested messages for a more generalized protocol:

• WM_ISCONTESTSUPPORTED - query if a given contest is supported

    Syntax: WM_COPYDATA WM_ISCONTESTSUPPORTED <contest-key>
      where
        contest-key = unique contest identifier from column 1 of N1MM+'s
                      Select Contest dialog (e.g. CQWWCW, CQWPXCW, FD, NAQPCW).

This first message will allow N1MM's Morse Runner Setup dialog to query
Morse Runner (once started) to verify that the requested contest is supported.
This allows N1MM to support most contests supported by Morse Runner.

Unfortunately, Morse Runner must be started before asking this question.
Perhaps we could use a configuration file to communicate which contests
are supported by Morse Runner (e.g. file read by N1MM contains MR-supported
contest keys).

• WM_SETCONTEST - set a given contest

  Syntax: WM_COPYDATA WM_SETCONTEST <contest-key>
    where
      contest-key = unique contest identifier from column 1
                    of N1MM+'s Select Contest dialog
                    (e.g. CQWWCW, CQWPXCW, FD, NAQPCW).

This message will select the current contest using and set a
default exchange value. WM_SETMYEXCH must be called to set the
contest-specific exchange.

Question: How do we return an error string for an invalid contest key?

• WM_RUNPILEUP - start selected contest in Pile-Up mode

    Syntax: WM_RUNPILEUP

This message will start the selected contest in Pile-Up mode.
It is called after the WM_SETCONTEST and WM_SETMYEXCH messages.

• WM_RUNSINGLE - start selected contest in Single-Call mode

    Syntax: WM_RUNSINGLE

This message will start the selected contest in Single-Call mode.
It is called after the WM_SETCONTEST and WM_SETMYEXCH messages.

3. To generalize all this into a single message, we could extend the
   WM_SETCONTEST with a compound argument string containing both the contest
   key and contest exchange. This extension will replace WM_SETMYEXCH.

There are two options:
a) using a semi-colon to separate and :

    WM_RUNCONTEST str='<contest-id>; <sent-exchange>'

Examples:

    WM_SETCONTEST str='CQWWCW; 3'
    WM_SETCONTEST str='ARRLFD; 3A OR'
    WM_SETCONTEST str='NAQPCW; Mike OR'
    WM_SETCONTEST str='SSCW; A 73 OR'

b) using name-value pairs:

     WM_SETCONTEST str='key=<contest-key>;exchange=<sent-exchange>[;...]'

Examples:

     WM_SETCONTEST str='key=CQWWCW; exchange=3'
     WM_SETCONTEST str='key=FD; exchange=3A OR'
     WM_SETCONTEST str='key=NAQPCW; exchange=Mike OR'
     WM_SETCONTEST str='key=SSCW; exchange=A 73 OR'

The advantage of this second form is that additional name=value pairs can
be added in the future without changing protocol or adding new messages.

[w7sst]

Hi Steve @n2ic ,
Thanks for joining as a collaborator to this repository. We can continue our discussion on the N1MM-MR communication protocol here so we can keep a history of the conversation. Welcome aboard!
73, Mike

[n2ic]

Trying to run from N1MM for the first time. Here's the debug output. I'm stuck, wondering what SetMyExch1 really wants. Is there a way to direct the assert output to the debug.txt file ? Right now, I can only guess that it's stopping at the assert.

TMainForm.FormCreate: GetCurrentThreadID 55960
FromIni
TMainForm.SetPitch(3): Pitch 450, Tst.Modul.CarrierFreq 459.38
TMainForm.SetBw(3): Bandwidth 250, Filt.Points 31, Filt.GainDb 3.01
LoadCallList
reading Calls & Zones from C:\Users\Steve\Documents\TeamFoundationServer\N1MM Logger on .NET\main\N1MM Logger.Net\bin\Debug\SupportFiles\mr_db.txt
TArrlFieldDay.Create
LoadFdHistoryFile
Calling slst.LoadFromFile('C:\Users\Steve\Documents\TeamFoundationServer\N1MM Logger on .NET\main\N1MM Logger.Net\bin\Debug\SupportFiles\FD_2021-008.txt')
arg1 = -h
arg2 = -r1
SetContest(scCQWW, '')
ConfigureExchangeFields(etRST, etCqZone, '')
WM_SETCWSPEED: str=32
WM_SETMYCALL: str=N2IC
TMainForm.Edit4Change
WM_SETMYZONE: str=4
TMainForm.SetMyZone('4')
WM_SETMYEXCH: str=4
WM_SETACTIVITY: str=4
WM_SETDURATION: str=10
WM_PITCHSET: str=450
TMainForm.SetPitch(3): Pitch 450, Tst.Modul.CarrierFreq 459.38
WM_SETCONTEST: str=CQWWCW
TMainForm.SetContestByKey('CQWWCW', '4')
Ini.FindContestByKey('CQWWCW') --> scCQWW: True
SetContest(scCQWW, '4')
ConfigureExchangeFields(etRST, etCqZone, '4')
TMainForm.SetMyExchange('4')
SetMyExch1(etRST, 4)

[w7sst]

It looks like the WM_SETMYEXCH: str=4 (from N1MM) is sending the expecting short form only. This is good. The bad part is that my code is expecting 5nn 4, not 4.

In the email I had sent earlier, I showed...

WM_CQWW: starting cqww
  TMainForm.SetContestByKey('CQWWCW', '5nn 3')
    Ini.FindContestByKey('CQWWCW') --> scCQWW: True
    SetContest(scCQWW, '5nn 3')
      ConfigureExchangeFields(etRST, etCqZone, '5nn 3')
        TMainForm.SetMyExchange('5nn 3')
          SetMyExch1(etRST, 5nn)
          SetMyExch2(etCqZone, 3)

The strings above show the correct strings for ConfigureExchangeFields and SetMyExchange. My code is currently expecting 5nn 4. Your code is sending in only 4.

As a workaround in my code, before calling SetContest() you can insert '5nn' to get this to work. This would be for scCQWW and scWpx, not scFieldDay (FD will send in both strings, 3A OR).

[n2ic]

Okay...made the change so N1MM sends 5nn 4. Working fine now for CQWW. I attached debug.txt so you can look at it and makes sure it's not "accidentally" working.

debug.txt

[w7sst]

Hi Steve,
It is really great to see this starting to work.

Instead of adding the work-around in N1MM, let's put the workaround in MR so I can fix this problem later. See Issue #42 for more information. I will get to this sometime this fall.

Please insert the following into your code before the call to SetContestByName():

                      // work around - insert '5nn' into exchange. We want
                      // N1MM to send its exchange setup field (i.e. only '4').
                      // Mike will fix this in MR (see Issue #42).
                      if (ContestKey = 'CQWWCW') and not SentExchange.StartsWith('5nn', True) then
                        SentExchange := '5nn ' + SentExchange;

Here is the larger context showing the message handler:

                    DebugLnEnter('WM_SETCONTEST: str=', str);
                    assert(not SentExchange.IsEmpty,
                      'SETMYEXCH must be called before WM_SETCONTEST');
                    if Ini.IsContestSupported(str) then begin
                      ContestKey := str;
                      // work around - insert '5nn' into exchange. We want
                      // N1MM to send its exchange setup field (i.e. only '4').
                      // Mike will fix this in MR (see Issue #42).
                      if (ContestKey = 'CQWWCW') and not SentExchange.StartsWith('5nn', True) then
                        SentExchange := '5nn ' + SentExchange;
                      Mainform.SetContestByKey(ContestKey, SentExchange);
                      Result := 1
                    end 

[w7sst]

@n2ic Hi Steve,
Have you been able to make anymore progress on this? Last I heard, you had my code running with a few modifications made to N1MM.

I've been slowly coding away on support for ARRL DX Contest.

73,
Mike