Sayma v2.0 production and testing

[jbqubit]

To make it easier for Sinara community to follow Sayma progress let's put high level discussion of hardware production and testing timeline in this Issue. As usual create independent Issues for specific test results and problems.

Here's the latest as I understand it.

• 8/14 One Sayma_AMC is stuffed. Testing started at Creotech.
• 8/26 One Sayma_RTM is stuffed.

@gkasprow @marmeladapk

[hartytp]

@jbqubit is there a definitive list of the gateware/software support that was funded and the relative priorities? Eg was 1gsps operation funded in the end, and where does that lie in the priorities list?

@marmeladapk when do you think you can ship a board to m-labs?

[gkasprow]

@marmeladapk told me a few days ago that Sayma AMC passed initial tests and they are working on bringing up other boards. I'm out of the office so don't know the exact status.

[jbqubit]

@hartytp The contracts are detailed here.

Hardware solicitation W911QX-19-Q-0060
https://www.fbo.gov/index.php?s=opportunity&mode=form&id=9f270a64ff963bf91530947b5449845b&tab=core&_cview=0

Software/gateware solicitation W911QX-19-Q-0166
https://www.fbo.gov/index?s=opportunity&mode=form&id=6d0280e7265cd24ebbd85e1dc1b06fdc&tab=core&_cview=0

Details you asked about are in Extended Deliverable C.3.2.2.1 of the second solicitation.

@akaminska

[hartytp]

Thanks @jbqubit (although, I think you got the hw/software contracts the wrong way around there).

Am I right in interpreting that as saying that the first priority (after getting the bring up working and RF output reliable) is phase synchronisation. After that, 1GSPS operation is next?

[marmeladapk]

We started testing Sayma AMC. So far power, I2C and SDRAM were tested. We're waiting for Sayma RTM to test GTx links and other functionalities. After that we could send one Sayma AMC to @sbourdeauducq. Then after we verify that RTM is working correctly we could send RTM.

[hartytp]

Great

[akaminska]

For everyone curious out there @sbourdeauducq - the first two Sayma RTM boards are out of production and we started working on them already. More details soon.

[sbourdeauducq]

@akaminska @marmeladapk Any update on the RTM testing?
Also, assuming the tests look ok, when are you planning to stuff more boards?

[marmeladapk]

@sbourdeauducq

This Wednesday I finished checking power supplies on Sayma RTM (tedious process, when there are many of them). Wojciech ran parts of test suite and so far DAC JESD PRBS is working on 4GBPS, today we'll know if 10GBPS is working.

On AMC we checked that it gets power from crate (openMMC).

Things that we are planning to do before sending one pair of boards to @sbourdeauducq:

• JESD @ 10GBPS
• Test signal output from DACs
• PORT0 in crate
• flash memory
• AMC+RTM in crate (if we have overcurrent on 3V3MP).

We plan to green-light further production this or next Monday depending on progress. Next week we should also get Basemods.

However I have bad news concerning Metlino as PCB production is delayed. I cannot tell right now when it will arrive (>1month?).

[hartytp]

That's great! Good work @marmeladapk

[sbourdeauducq]

IMO Metlino delays are less important since we can use Kasli instead, but Sayma has features that aren't present in any other system.
Do you know when more Sayma boards will be stuffed?

[jbqubit]

Both Hardware Developer and Software/Gateware developer contracts share specifications and terminology. I created a wiki page to keep this sorted.

I created several Issues for Hardware Developer tasks based on the contract. @marmeladapk please create additional tasks corresponding to the remaining deliverable items.

@sbourdeauducq asks

Do you know when more Sayma boards will be stuffed?

I don't expect that the will be stuffed until the first round of stuffed boards pass testing. But that's at the discretion of @marmeladapk. However the contract is explicit that the Hardware Developer is to complete baseline testing (HT3 #122) prior to prototype shipment. This workflow is designed to maximize liklihood that hardware bugs are found and patched prior to shipping.

[jbqubit]

IMO Metlino delays are less important since we can use Kasli instead

Best if Metlino not delay shipping of Sayma prototypes. Testing Sayma using Kasli core device is a fine way to boostrap the future all-uTCA Metlino+Sayma system.

[marmeladapk]

Basically the only issue that is preventing us from sending Sayma to @sbourdeauducq is #126. We tested JESD @ 10 GBPS and PORT0 @ 4 GBPS. We postponed testing DAC output until we can figure out flash issue.

Other than that, AMC+RTM have problems with powering up in the crate, but we'll debug it after flash (AMC powers up fine, however I'm not sure that it's overcurrent since I checked that RTM consumes only around ~10 mA on 3V3MP).

[jbqubit]

The work plan is structured so that other labs/contractors receive hardware only after thorough testing is complete at Creotech. Recall that HT3 calls for both thermal tests (in TS7) simulation of heavy loading (in TS2) prior to shipping PCBs to other labs.

[marmeladapk]

@jbqubit First photos, that we have of these boards (before any hardware changes).

[hartytp]

🎆 cool!

[jbqubit]

Splendid.

[jbqubit]

What's the status of Metlino? On 9/6 word was it would be a month.

[jbqubit]

By email @marmeladapk said "I don't anticipate that we'll be ready to send Metlino until December." Is M-Labs is OK with proceeding with tasks including Sayma-Sayma synchronization using Kasli as Core Device until Metlino is available?

[sbourdeauducq]

Yes, either works.

[jbqubit]

@marmeladapk What's status of BaseMod v2.0 and TestMod v2.0? How's the testing coming along?

[jbqubit]

@marmeladapk Ping. Please let us know the status.

[marmeladapk]

Sorry for the delay, I wanted to catch up with one of the developers.

All BaseMods are on my desk and are waiting for me to test them.

Stress test is complete and we didn't find any issues.

Additionally I fixed one issue that disabled crate power when AMC+RTM were starting. Then I was able to start 7 Saymas+RTMs in a small crate.

Clip of all Saymas starting up in crate

What still remains for us to do:

• We're still waiting for panels for RTMs. Once we get them we can implement RTM hotswap.
• Wojtek is still working on DAC output with AD IP-core. Once we have that we'll be able to finish all tests in TS7
• We probably need to add a workaround to MMC when crate is starting up and all hotswaps are inserted. Only two cards get their power. Probably too much traffic on I2C lines for MCH, but we'll investigate further.

[jbqubit]

NAT power supply supports staggered power-on for AMC slots. Have you tried using that feature?

Seems odd that the LED load indicator on NAT-PM-AC600D doesn't change as hot swap handles are pushed in. Guess that must be without thermal dissipation with TestMod_FMC resistors?

When can you ship TestMod and BaseMod to M-Labs?

[jbqubit]

ping @marmeladapk

[marmeladapk]

Update:

We need to write support for:

• attenuators on basemod
• RF switch on basemod
• RTM thermometers
• MGT clk output
• RTM hotswap

All of these should be easy. After we finish this we'll do all tests from HT3 which require DAC output + test all boards. I hope we'll be able to distribute them before holidays.

[jbqubit]

@marmeladapk When we spoke on 11/27 you said:

• AMC and RTM front panels had been ordered but not yet arrived. What's the status?
• All AMC are stuffed (6 total) but not all were tested. Testing was to happen in early December so that if problems are found they can be addressed in parallel with DAC problem. What's status of testing?

[jbqubit]

@marmeladapk Ping.

[marmeladapk]

• We have further delay with Sayma AMC and RTM panels due to manufacturer's destructive mistake. We hope to have new panels before the end of January (ordering new panels from schroff+finding new manufacturer+holidays).
• All AMC and RTM cards were tested. One AMC card reported problems with P5V0. I didn't have time to investigate it yet. However we have 6 pairs that passed tests.

[sbourdeauducq]

* All AMC and RTM cards were tested.

Did you get any RF out with ARTIQ?

[marmeladapk]

@sbourdeauducq No, DAC setup on ARTIQ now fails on checking SYSREF margin (I'm speaking from memory now, I cannot remember the exact message now). However I checked that SYSREF phase was changing during this test with oscilloscope. If you want we can try to debug it (and your card) and discuss it in #141.

[sbourdeauducq]

We can test SYSREF/sync and RF output separately.

[jbqubit]

From call with Creotech this morning. Several Issues remain for HT3.

• One DAC doesn't work on M-Labs test system. DAC testing #141
• BaseMod... one channel is lower amplitude (Issue)
• NAT NativeR5 power-up with all slots filled... only 2 Sayma startup...
  No power to some cards in crate #146
• NAT NativeR5 crate... some boards can't communicate with MCH Red LED on MCH in some crate slots #147
• documentation of Metlino production manufacturing and testing status Metlino#60 (comment)
• HT4 serial no, wiki page for tracking PCB rework  #123

Front Panel (FP) status

• found vendor with one week turn-around
• received 1 AMC FP, problems corrected
• will receive 1 RTM FP next week
• no status on BaseMod FP (Issue)
• no status on Metlino FP

[marmeladapk]

@jbqubit All basemods have frontpanels moutned, however without any markings.

[jbqubit]

@marmeladapk When we spoke on Friday you indicated that the Sayma and Metlino front panels would arrive at Creotech on Monday. Did that happen? Can you now ship Sayma to M-Labs?

[jbqubit]

@marmeladapk asked that I post the contract docs to the wiki as a reminder of the division of labor for Sayma v2. They are here.

[jbqubit]

Creotech shipped the following to M-Labs:

• 1 AMC and 1 RTM with front panel
• 1 AMC and 1 RTM with no front panels

Creotech shipped the following to Oxford:

• 3 AMC and 3 RTM with no front panels

@hartytp @sbourdeauducq Please indicate when you've received them.

[sbourdeauducq]

Received all.
I still want to swap at some point the Sayma boards without front panel and with 1 DAC not working for boards with front panels and both DACs working.

[jbqubit]

Please edit PCB modification wiki page to indicate for which board "1 DAC not working."

[hartytp]

Received all

[jbqubit]

@marmeladapk When can you ship 2 full Sayma to M-Labs? I understand @sbourdeauducq will return to you his broken board upon receipt. Sayma-Sayma synchronization is not yet been properly tested so this is a high priority.

m-labs/artiq#795 (comment)

[hartytp]

@jbqubit what's the point of worrying about Sayma->Sayma synchronization when FPGA->DAC synchronisation does not yet work reliably on a single Sayma? Surely the bottleneck here is someone looking into FPGA->DAC sync (@sbourdeauducq are there plans to get back to that?)...

[jbqubit]

@hartytp Agreed that there getting FPGA-DAC synchronization to work is the first step. M-Labs needs a pair of Sayma to validate FPGA-DAC synchronization via DRTIO in a multiple-Sayma context. A pair of Sayma is also needed for later analog synchronization tests. It takes a week or more to ship hardware to HK.

@marmeladapk When can you ship 2 full Sayma to M-Labs?

ping @marmeladapk

[akaminska]

Hi Joe, We are working on shipping the Saymas. Logistics is not easy when all R&D is on home office. The current plan is that a mechanical engineer will come in to the R&D office on Monday to fit the panels (they arrived, but need a few adjustments). On Tuesday we will have a possibility of transporting goods from our R&D office to our main office and warehouse for packing and preparing for shipment. On Wednesday we should be able to ship the package. Best wishes, Anna

…

Wiadomość napisana przez Joe Britton ***@***.***> w dniu 02.04.2020, o godz. 17:12: @hartytp <https://github.com/hartytp> Agreed that there getting FPGA-DAC synchronization to work is the first step <https://github.com/sinara-hw/Sayma_AMC/wiki/Sayma_v2_contract_m-labs#c301-dac-fpga-synchronization>. M-Labs needs a pair of Sayma to validate FPGA-DAC synchronization via DRTIO in a multiple-Sayma context. A pair of Sayma is also needed for later analog synchronization tests. It takes a week or more to ship hardware to HK. @marmeladapk <https://github.com/marmeladapk> When can you ship 2 full Sayma to M-Labs? ping @marmeladapk <https://github.com/marmeladapk> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#121 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK5FLFLXUOE26BMQZJBBJG3RKSTONANCNFSM4IQLBRZA>.

[jbqubit]

@akaminska OK. Thank you for the update.

I'm annoyed that this wasn't done a month ago. You guys did great work on Sayma v2. I want to be sure that M-Labs has an opportunity to work with a fully functional board pair.

[jbqubit]

@akaminska said via email to @sbourdeauducq ...

We are preparing for you in the first half of next week a package which will contain two pairs of Sayma AMC + Sayma RTM (we will ask you later to return the malfunctioning Sayma AMC + Sayma RTM pair that you have at M-Labs now), 1 Metlino and 2 BaseMod, all with panels. We will include a set of components required to integrate Metlino and MCH and prepare a documentation showing how to perform this process.

On a separate note, we will also include a Kasli v2 in the package.

@sbourdeauducq asked

What about Metlino, is it ready?

I agree with @sbourdeauducq. Do please also send a Metlino to M-Labs.

[akaminska]

As I mentioned in the e-mail you quote, we are sending 1 Metlino to M-Labs as well :)

…

Wiadomość napisana przez Joe Britton ***@***.***> w dniu 06.04.2020, o godz. 19:05: @akaminska <https://github.com/akaminska> said via email to @sbourdeauducq <https://github.com/sbourdeauducq> ... We are preparing for you in the first half of next week a package which will contain two pairs of Sayma AMC + Sayma RTM (we will ask you later to return the malfunctioning Sayma AMC + Sayma RTM pair that you have at M-Labs now), 1 Metlino and 2 BaseMod, all with panels. We will include a set of components required to integrate Metlino and MCH and prepare a documentation showing how to perform this process. On a separate note, we will also include a Kasli v2 in the package. @sbourdeauducq <https://github.com/sbourdeauducq> asked What about Metlino, is it ready? I agree with @sbourdeauducq <https://github.com/sbourdeauducq>. Do please also send a Metlino to M-Labs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#121 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK5FLFMIEKHRVESWUYIGCCTRLIDVHANCNFSM4IQLBRZA>.