Day 481: Part samples and drop tests
over 7 years ago – Sun, Oct 23, 2016 at 05:15:36 AM

 Hello from Oakland!

TL;DR: Keycap tooling is almost done! The feet design we were concerned about last time was as bad as we thought. We think we’ve got something much better. We’ve made some tweaks to the electrical design and firmware; we’ve figured out some stuff about keycap painting and kitting; we’ve been doing drop tests. In general, manufacturing is proceeding apace: slower than we’d like but moving forward steadily. The changes to the base/foot design potentially adds 3-4 weeks of delay. We’re currently betting that the first units will start to ship from the factory in November.

Keyboardio & Son

It's been quite a month. Jesse's just back from two weeks in Shenzhen working with our factory and Kaia's hard at work starting to figure out shipping logistics.

We've tried hard to be as transparent as we can about the process of getting your keyboards made, but haven't talked a whole lot about what else we've been up to.

At a dinner party a few weeks ago, we were chatting with friends and mentioned that we were getting ready to have a baby in late November. A friend replied "oh yeah, you guys are doing that keyboard thing."

While we certainly think of the Model 01 as our baby, we were actually talking about a real baby.

So, yeah, sometime around the end of November, we're going to be parents.

As of right now, it's not entirely clear whether the baby will arrive before or after the Model 01. We're working hard to get the Model 01 shipped as soon as possible, but babies have their own schedule.

This will, of course, put a bit of a squeeze on the amount of time that Jesse can spend in Shenzhen, so he spent a good part of his most recent trip making sure that we have a solid plan for resolving the remaining production issues with the factory and that our team at HWTrek is set up to manage the process in China on our behalf.

Depending on how things work out, Jesse may squeeze one more trip to China in toward the end of October.

(And no, we haven't decided whether we're going to inflict QWERTY on the kid. We figure we've still got a couple years before we have to make that call.)

Manufacturing overview

As we've talked about before, our factory is a smaller keyboard and mouse manufacturer. One of the reasons we chose them over trying a larger factory again is that we're "right-sized" for them. While our order isn't big enough to make a big factory pay any attention to us, we're a commercially significant customer to a smaller factory. The upside of that is that they'll be more willing to work with us to resolve issues. The downside is that they don't necessarily do as much in-house as a larger factory. In our case, this has meant that they've had to reach out to partners for some of the EE debugging we've needed and that most of our injection molding is being done by partners. The factory has an injection-molding shop in house, but decided to farm out much of the injection mold making and the actual injection molding.

When we asked the factory why they decided to outsource something they can do in house, they told us that some of our design requirements are...a little out of spec for a regular keyboard factory. In particular, we've picked higher-than-usual quality plastics for some parts and have a few parts with relatively complex injection mold designs.

The outsourcing has led to a few frustrating delays and a bit of miscommunication, but we're pretty sure that it's going to result in a higher quality keyboard.

Bottom plates

We mentioned in the last update that the factory had decided to start making injection molding tooling for all the plastic parts of the Model 01, even though we hadn't signed off on it yet. They did this to try to accelerate the production schedule, telling us that they were pretty confident that we'd be able to accommodate any design fine-tuning without having to remake the tooling from scratch.

The downside of making and modifying tooling like this (as we understand it) is that modified tooling won't last as long as tooling that was done right the first time. That said, the "regular lifetime" for tooling like this is...a few hundred thousand keyboards. If we run into tool lifetime issues with the Model 01, we're going to be very, very happy campers.

Talking to friends who have experience with large-scale manufacturing, this revelation resulted in gasps of surprise. "Nobody does that. That's weird." Talking to friends who've done small-scale manufacturing in China, we got a somewhat different response. "Oh yeah. That happens. It's annoying, but as long as the factory agrees that they're on the hook for fixing issues, it's not a big deal."

To be clear, any time you make injection molds, there will be fine tuning. You make the tooling and try it. You look for things like sinkholes, warping, or other plastic flow issues. And then you tweak the tooling. The issues we were worrying about were things like "what happens if the design for the feet turns out to be unworkable?"

Upon arrival in Shenzhen, Jesse got to check out samples from the first and second versions of the bottom plate tooling. The factory had already found and fixed a number of small flaws in the tooling.

 At the end of the first week, "T3" samples of the base plates showed up from the factory handling the baseplates. These samples were white, though the final version will be black.

One of the things we were worried about was the tripod mounts. These screws need to be stuck into the plastic pretty tightly or they're pretty much worthless. Early prototypes from the factory had the tripod mounts glued into the base plates. Once we got back the T3 samples, Jesse tried hard to rip one out...and only succeeded once he cut a base plate in half and used shears to rip the plastic away from the tripod mount. They shouldn't be going anywhere. Now that we have injection mold tooling, the bottom plates are actually being molded around the brass tripod mounts. There is video somewhere of Jesse trying hard to rip out a tripod mount, hurting his hand doing so and grinning stupidly, signing off on that aspect of the design.

We don't have pictures of how injection molds with inserts work, but you can likely find something similar on YouTube.

The bottom plates also include features to protect the USB Type C and RJ45 jacks. Those bits seem to work pretty well, too.

As you'll read in a moment, the bottom plates will need to be modified to remove mounting brackets for the flip-out feet. After that, the mold-making factory will add some pleasant texture and a Keyboardio logo to the underside of the plates.

The Feet

We mentioned in the last update that the design for the feet wasn't going to cut it. This was...rather frustrating. The factory wasn't sure whether this was a design flaw or something that would go away once we got to try the real injection-molded base plates, feet and rubber pads.

In Shenzhen, we tried the real injection-molded part. The problem did not go away. So we started to look at fixes.

The simplest solution would be to move the feet further toward the outer edges of the keyboard. If the keyboard were a normal, boring, rectangular shape, that might even solve the problem, but between the Model 01's butterfly shape and our desire to support a combination of tenting and tilting configurations, it wasn't going to be workable without providing you with a bunch of feet of different lengths to swap out to get the angle and configuration you'd want. Even then, it wasn't going to be a great experience. And there would be a bunch of small parts to lose.

As we were considering whether we could relocate the flip-out feet, the factory's R&D team introduced us to what we're told is a mechanical engineering aphorism: "The screws are always in the best locations. If you need to add a feature, they'll be in the way." From this, they spent some time looking at moving to screw-in feet on top of the existing screw holes. This worked a little better than moving the flip-out feet, but shared most of the same problems.

The next morning, the mechanical engineer was off-site attempting to expedite things with one of the tooling suppliers and Jesse started to fiddle around with an idea that had been bouncing around the back of his head: a pair of angled stands using the tripod mounts. By the time someone noticed, he'd hacked together something that almost worked using a screw, spare pieces of ABS plastic and superglue.

Borrowing some modeling clay, Jesse introduced the ME to the concept of "shitty prototyping." The basic idea is that a prototype should be just good enough to test the one thing you're trying to test. It doesn't matter if it looks silly or uses the wrong materials or falls apart in 10 minutes. The first prototypes of the new stand were...pretty shitty, but we quickly got closer and closer to something workable that provides much, much better stability for your Model 01.

The design we’ve ended up with: a pair of identical stands, just about the same size as the Model 01 itself, that screw into the tripod mount on the bottom of the Model 01. You can rotate them to get various tilt and tent angles. We will ship the Model 01 with these stands as well as the two center bars we’ve talked about in previous updates.

Supporting all of the tent and tilt configurations we want for the Model 01 is a deceptively complex engineering challenge. At this point, we've probably run through more designs for the feet / stand than any other aspect of the keyboard's design.

We ended up needing to slightly relax one of our requirements: If you want to use the provided stand to tent and tilt the keyboard at the same time, the two halves can't be connected by the center bar. In practice, just about everyone who wants to tilt and tent the Model 01 at the same time would prefer that the two halves be independently adjustable, so we don’t think this change is likely to cause too much trouble.

If you want to tent and tilt the keyboard at the same time, the two halves can’t be connected with a center bar.

If you’d prefer to keep the two halves of the keyboard together with a center bar, you can use the stands to either tent or tilt it.

The other tradeoff is that the stands, while lightweight and detachable, are a bit bulkier than integrated feet would be.

The last prototype stand that we built before Jesse came home from Shenzhen was just a little bit too wide and a couple of centimeters too shallow. We're working with the factory to refine the shape of what we hope will be the final prototype before we sign off on the design and they get to work on the injection mold.

 One nice thing about this design is that it'd be relatively easy to build your own stands to get any combination of tent and tilt that you want.

Enclosure

While in Shenzhen, we met with the factory who will be milling our enclosures. The big open issue with them is the screw inserts we specified for holding the enclosure to the base plate. In their experience, the knife-edge style inserts we specified tend to cause too high a wooden part failure rate during insertion. The part they counter-proposed doesn't have enough grip to make us happy. We've suggested a third alternative, that they're trying to source now.

They also presented us with our two options for wood finish: high-gloss and matte. We talked it over and ended up agreeing on the matte finish. It looks classier and doesn't show fingerprints like the glossy finish. We went back and forth a few times with them because we really wanted to understand their wood treatment processes. It took a bit, but they sent over all of the lab test reports from their finish provider. Their main wood finish is polyurethane. We knew they were using a solvent along with the PU. When we asked what the solvent was, the answer we got convinced us that Google Translate was broken. "Banana Oil." Turns out that Banana Oil is totally a thing. The chemical name is isoamyl acetate. In addition to being a solvent for varnishes and lacquers, it has a strong banana scent and is used to add banana flavor to food. You can read about it on Wikipedia. In the end, our biggest disappointment about the finish is that by the time the varnish dries, it doesn't have even a hint of banana scent.

Center bars and rails

One of the other things that we got to test in Shenzhen was the interconnect mechanism. The flat and tented slide bars both hold the keyboard together well. Picking up the keyboard with one hand shows a little bit of flex, but it takes real work to get the two halves to disconnect by shaking one half of the keyboard up and down. Jesse was able to snap the two halves of the keyboard apart by gripping them with two hands and bending, but it was really difficult. Despite a loud noise when snapping apart, the center bar and rails showed almost no deformation and slid back together solidly. Plastic is not without its advantages.

The one problem we discovered was that after a good deal of flexing and a few nice sturdy one meter drops onto concrete, the rails developed a bit of wiggle. The original design had the rails attached to the bottom plate with tapping screws. When tapping screws are screwed into something that gets a bunch of stress put on it, they lose a bit of their grip. The factory has said that they're going to replace the tapping screws with machine screws and inserts similar to what we're doing for the tripod mounts. At the same time, they're going to reverse the direction of the guide arrows on the rails, as they're currently backwards.

The factory tried center bars made out of a Polycarbonate/ABS mix and bars made out of glass-filled nylon. Both seem to hold up well, but the glass-filled nylon bars seem to hold up just a bit better.

The factory needs to update the tented slide bar to remove the built-in feet. With the new stands, they no longer have a purpose. That'll also make the tented slide bar much thinner. They'll also put some decoration on the bottom of both center bars.

Keycaps

We mentioned in an earlier update that we'd signed off on the design for the keycaps and that the factory was having tooling made. Keycap tooling is the most expensive single purchase as we're making the Model 01. It's a pretty fiddly multi-cavity mold. (That means that they've put all 64 keycaps in a single injection mold.)

When Jesse got to Shenzhen, he saw samples from the T1 version of the keycaps and was told that the keycap tooling factory was hard at work on tweaking the molds to get us to T2. One of the biggest issues with the first version of the keycap tooling was that the factory had gone back to our keyswitch's datasheet as a reference for the size of the "stem" that holds the keycap onto the keyswitch, rather than measuring from actual sample keyswitches. The datasheet, while correct, is insufficiently precise to result in a good fit between keycap and keyswitch. The keycap factory also left out the key numbers the factory had designed into the underside of each keycap. To fix this, they ended up needing to weld and recut the inside of every single key.

The keycap factory wasn't able to finish the T2 keycaps by the date they originally estimated. When pushed for status, they sent an excellent powerpoint presentation walking through all the changes they were making and the current status of those changes. While delays are never fun, clear communication about delays and status is a truly wonderful thing.

On his next-to-last day in Shenzhen, Jesse was able to visit the keycap factory and saw our injection molds being recut on the EDM. 

He also got to check out keycaps being molded on the keycap factory's very, very automated injection machine. We've been told that our keycaps will be made on this machine with a similar level of automation.

One of the neat things about how the keycap molds were designed is that the keycaps are connected to the sprue (the channel that plastic flows through the injection mold) on the bottom edge, rather than in the middle of the part. This means that the keycaps won't have an unsightly dot in the middle of one of their sides.

 We're just starting to review the T2 keycaps to figure out what issues still need to be corrected.

 The keycap factory also showed us the trays our factory will be using to paint the keycaps. The trays keep the keycaps stable as they go through the painting, laser engraving, and UV clear coating process. Each tray can be reused four or five times before it's discarded.

Key labels

Last month, we asked you to fill in a survey indicating what sorts of alternate key labels you'd want if we could pull them off. We'll keep the survey open for another week or two. If you haven't filled it in yet, you can find it here: https://keyboardio.typeform.com/to/Zh86UM

Please note that this survey is to help us gauge interest. By filling it in, you're not committing to anything. By skipping it, you're not missing anything.

Right now, we're looking at a minimum order of about 250 sets of a given set of labels. We're also looking at a retail price of about $40/set for those keycaps. That price may go up or down a bit, but hopefully not by too much.

As of now, the top five non-QWERTY answers are:

• Translucent blank 
• Black paint with translucent circle 
• Dvorak 
• Unique symbols or runes on each key 
• Black paint, blank

After that, it's a pretty dramatic fall-off. The other options, in order of popularity are: Colemak, Workman, Other, QWERTZ, Scandinavian, BEPO, AZERTY, Malt. Some of the "other" options included Neo, Cyrillic, Spanish, Hebrew, and UK QWERTY.

In the next couple weeks, we should be reaching out to folks who offered to help figure out how all of these layouts should actually map to the Model 01’s keys.

Drop tests

We had an exciting afternoon at the factory in Shenzhen, where we ran through a variety of one meter drop tests. The first tests were dropping onto cardboard + thin foam approximating a carpet. The Model 01 fared quite well dropped onto “carpet” from a variety of angles.

Next up were tests onto bare concrete. When we dropped just the wooden enclosure on its edge from a height of one meter onto a concrete floor, it did exactly what you'd expect: It split in half. Happily, the plastic bottom plate, circuit board and metal keyplate give the Model 01 more structure than the wood enclosure has on its own. When we simulated pushing the whole keyboard off a table onto a concrete floor, keycaps popped off, but there was no visible damage. When we dropped a fully assembled keyboard on its edge from a height of one meter onto a concrete floor, the enclosure cracked, but did not shatter.

(Note: This video is very flickery because it was filmed in slow motion under bad fluorescent lights.)

It should go without saying that when you drop expensive combinations of wood and electronics onto concrete floors, there's a danger of bad things happening. We strongly recommend you not drop your keyboard onto a concrete floor, though we are planning on offering replacement wooden enclosures for sale at a non-predatory price.

After completing the drop tests, Jesse decided to see if he could break the keyboard by sitting on it. That was a mistake. The keyboard was fine, but Jesse was a bit sore. Do not sit on your Model 01.

Electrical

When we last wrote, we were still tracking a weird temperature issue with the NXP PCA9614 differential I2C controllers we're using. Just before Jesse got on the plane to China, we heard back from NXP that they'd had their engineering team take a look at our boards and, while the temperature was looking a little high, it wasn't abnormally high and shouldn't have any significant impact on chip lifetime or performance.

Next up was a slight buzzing noise emanating from our Chinese-made boards when the LEDs were showing a rainbow effect. Embarrassingly, this isn't something we'd noticed at all. @scanlime, however, managed to pick it up over the sound of a running fan, noise from the BART, and a couple of low quality CF light bulbs… by listening to a video we posted to Twitter.

Our US-made boards didn't make a peep. The Chinese boards nominally had the same components on them, but a slightly different PCB layout. Our first guess, that they'd been using off-brand capacitors, turned out to be a red herring.

Once Jesse got to China, we pursued a fix and trying to find the root cause in parallel. The factory's EE and EE consultant took a stab at trying to reduce the board noise by adding additional capacitance to the board. Sadly, it didn't really help.

The next guess was that the LEDs they were using were themselves causing the issue, although we were seeing it with LEDs from at least two different manufacturers. (There are multiple companies that make "APA102C" LEDs. Go figure.) The factory brought in a team from one of the LED vendors. One of the folks sitting across the table from Jesse was introduced as the guy who designed the IC inside the LEDs we were using. He took a quick listen to the boards and said "Oh, that's our old IC. I could totally believe that it's making noise. You should use our new LEDs. They're better. They're the same size, but have nicer specs. The only thing is that their pins are laid out differently. So you'll need to change your PCB layout. Oh, also, they speak a different wire protocol. So you'll need to rewrite your code." We explained that we'd look at their new datasheet, but that what they'd described was… likely to cause us to reject their solution. After they conferred for a bit, they said that they could cook up some new LEDs that were compatible with the old ones but used their new IC by Saturday. (This was Thursday.)

Dumbfounded, Jesse smiled and nodded and said that sure, that sounded like a great plan and that we'd be thrilled to try their new LEDs. While they didn't end up delivering, Western friends familiar with Shenzhen said that that such things aren't all that uncommon and that yes, our order of a few hundred thousand LEDs was totally the kind of volume that a local manufacturer would build a custom LED or chip for.

Later, we swapped in genuine APA102C LEDs from the original Taiwanese manufacturer, APA Neon. The boards still made noise. So, it wasn't the bad chip posited by the chip designer from the clone vendor.

By pure chance, Jesse found himself sitting next to Eric Pan, Founder and CEO of Seeed Studio at dinner one night. While they were chatting, Jesse described the issue he was running into and Eric invited him to drop by Seeed's new office with his boards.

The next Monday, Jesse found himself on Seeed's doorstep. A few minutes later, he was describing the issue to Seeed's EE lead, Bruce. It only took Bruce a few minutes to find a solution that almost completely eliminated the noise. He popped off some of our capacitors and Ferrite beads, replacing them with heavier-duty tantalum capacitors and wire inductors.

After only a tiny bit of pleading, Seeed agreed to make us new quick-turn PCBs in 24 hours (Rather than their typical 48 hour turn.) Jesse spent his afternoon sitting in Seeed's lobby frantically updating our PCB layouts to hit a 5pm deadline. The next afternoon, at 5pm, we had new boards ready.

The factory assembled the new boards and the situation is much improved. If you’re not trying to induce seizures with the LED effects, the boards are silent. If you hold your ear to the PCB, there is a slight detectable buzz when you're bouncing the LED brightness around at high speed. It's almost completely inaudible, and will be further muffled when it’s inside the enclosure. Over the next week, the factory is continuing to test and tweak the PCBs. At the same time, we're putting the current design through its paces here. We need to be sure that when run within spec, the Model 01 doesn't make audible noise.

Firmware

In the past month, we've mostly been focused on the Mechanical and Electrical parts of the Model 01, though we've done some work on the ATTiny88 I2C bootloader we talked about last month. We need to spent a bit more time testing it out on real assembled keyboards, but it looks like you're going to be able to update the Model 01's ATTiny88 keyscanner firmware without needing to unscrew anything or use specialized programming hardware.

With luck, this will never actually matter for 99.9% of Model 01 users, but for those of you who want to experiment with new key debouncing algorithms or want to play really interesting games with the LEDs, it should be a big help. And, of course, if we find a bug in the ATTiny88 firmware, it won't be impossible to fix keyboards in the field. That definitely helps us sleep a bit easier. You can find the current state of the bootloader code here: https://github.com/keyboardio/attiny_i2c_bootloader

We still have a bit of work to do for the Model 01's shipping firmware. We've decided to play it a bit conservative with the default firmware - USB power will be limited to a nice, spec-compliant 500mA and the keyboard won't have some of the fancier features that we've talked about over the past year. We’ve built prototypes of everything, but aren’t confident that we’ll have everything as polished as we’d like by the time the factory has to flash the firmware onto the keyboards.

We'd rather be shipping a full-featured firmware with all sorts of whiz-bang magic on day one, but have decided that you getting a solid, working keyboard that gets better over time is going to be a much better experience than you getting a janky, unreliable keyboard that you toss in a drawer out of frustration.

Rather than shipping potentially buggy advanced features in the default 1.0 firmware, we'll be making them available in regular (100% free and opensource) updates as we're happy with the implementations. And heck, you might even end up building cool features we hadn't planned to include.

Heck, one of you already is. Gergely Nagy doesn't have a Model 01 yet, but has already been working on a "tap dance key" feature for the Model 01. You can find his code on GitHub: https://github.com/algernon/arduino-kbd-tap-dance. His most recent blog post on the topic is here: https://asylum.madhouse-project.org/blog/2016/10/03/balance/

The one big feature we will definitely be adding to the Model 01 before mass production is a special "factory test" mode. This will let the factory test multiple aspects of the Model 01. They'll be able to visually test the red, green and blue elements of each LED. They'll also be able to test the mechanical functionality of each key easily.

Every keyboard factory we've visited has a fairly similar process for testing the keys on a keyboard. They have some sort of Windows program that shows an illustration of a 104 key keyboard on a screen. As the assembly line worker runs their finger across each row of keys on the keyboard, the keys on the screen change color. If some keys don't light up on screen, they get a closer inspection. If, after an additional review, a key is determined to be misbehaving, the keyboard is sent back for further analysis and rework.

Because some of the Model 01's keys don't map to "normal" keys on a USB keyboard, the factory's usual manual testing procedure will fall short with the Model 01's regular firmware. The "factory test" mode will map each of the Model 01's special keys to some reasonable key on a 104 key keyboard, so the factory's test software behaves as expected. If we have the time, we'll probably add additional visual feedback by lighting up the LED under each key as it's lit.

Box and Kitting

Shortly before leaving China, Jesse sat down with our sales person and started to talk through what the Model 01 box is going to look like. It will almost certainly be a simple heavy-duty white cardboard box. (When we get to retail sales, we'll add a nice full-color sleeve with pictures of happy people typing productively on a Model 01.) Inside, you'll find your keyboard, screwdriver, cables and stands. You'll likely also find packets of desiccant and some free bubble wrap.

The keyboard will come with short and long RJ45 cables. The short cable is just long enough to connect the two halves comfortably when they're connected with the center bar. The long cable should be about 1 meter long. As of now, both will be black. The USB Type C to Type A cable included with the keyboard should be about 1.5M long. We're working hard to make sure that the Type C connector is a "right angle" connector, so that the cable runs off to the side, rather than straight back from the keyboard, but we don't have a confirmation from the factory that they've been able to source them just yet.

Schedule

The issues with the feet and the PCB noise have definitely set us back a bit, though the factory is working really hard to make up as much time as possible. We're hoping to have a final design for the stand by the end of next week. After that, we believe that the limiting factor on the schedule will be how fast tooling for that part can get made. The factory has quoted us around 4 weeks for similar tooling in the past, but hasn't made a commitment on this tooling yet. We're working really hard to get you your keyboards as soon as humanly possible. We have a strong incentive not to let things slip past November, but are slightly at the mercy of the factory's schedule.

<3 j+k

Day 453: Working through issues with sample parts
over 7 years ago – Thu, Sep 29, 2016 at 03:07:50 PM

Hello from Portland, Oregon!

TL;DR

• We finally got a full sample of the enclosure + feet + interconnect bars. There's a problem with reverse-tilt configurations. The factory is on it. 
• Want non-QWERTY keycaps? Take our survey!
• We’re headed back to China in ten days to check the first parts coming out of the injection molds.

We're in Portland through this Monday for XOXO, come say hi! We should have some stickers for you.

Shipping addresses

A few of you have written to tell us that you’ve moved and to make sure we have current addresses. While we’re always happy to hear from you, don’t worry about shipping address updates. We’ll be emailing everybody who backed us on Kickstarter (and everybody who preordered on our website) to get current addresses before shipping.

Mechanical

When we last wrote, we said that we expected to have a second, fully integrated sample of the Model 01 sent to us by the week of August 14. When the factory tested out that sample locally, they found some issues with the feet and interconnect bars and ended up deciding to update the design and build a new sample.

That sample shipped out to us on August 29 and we got it on September 2. When we opened the package, we found that they'd only sent us the wooden enclosure, the plastic bottom plates, feet and center bars. Keycaps and metal keyplates weren't included.

The keycaps aren't too big a deal, since we had a full set of the "final, except for one tiny tweak" keycaps from just before Jesse left Shenzhen. Without the keyplates, however, we can't combine the plastic and wooden parts with the PCBs the factory sent us a couple weeks ago to build a full working test unit. Without a working test unit, we won't sign off on the full, final design.

Ordinarily, this sort of a delay would mean that the schedule would have slipped dramatically, but our factory did something a little bit unorthodox in order to try to keep the schedule as close to their original estimates as possible—they started making all of the injection mold tooling weeks ago with the "near-final" design, since they were pretty sure that any required changes could be made on the fly later. Minor changes to tooling are not uncommon, but this opens them up to some risk if significant issues are found before we sign off on the tooling.

As we started to test the sample, we found a bunch of minor issues with a number of tent and tilt configurations and feet not quite touching in the right places. Most of these turn out to be traceable to a pretty simple issue: The nice round rubber bumpers on the bottom of the Model 01 are a custom part designed so that all four corners of the keyboard sit comfortably on the desk. The factory didn’t have those ready, so they made the sample with a similar pre-purchased part that’s not quite the right height. When we tried to simulate the correct design by sticking on some additional rubber feet, most of these small fit issues were resolved.

We weren't thrilled with the quality of the rubber stuck to the flip-out feet, but the factory clarified that it was also a purchased part and not what we should expect for a final design. They said they'd send us 3D CAD of the final design this week.

The tripod mounts on the bottom of the keyboard are nuts inserted into a plastic cavity. On the prototype, they're inserted and glued after CNC milling. On the production keyboards, they'll be inserted during molding, which in theory should make them sturdier. We had some questions about how sturdy they'll be in the face of… harsh treatment and the factory is going to talk with their injection tooling designers about adding additional mechanical reinforcement for the nuts.

There were a few small issues with the wood finish that we said we wouldn't be able to accept on delivered units. The factory told us that they'd make sure the wood supplier understood the issue.

One of the plastic feet was installed backward. It was an easy mistake—there were no markings on the front or back of the feet to indicate which side was the inside and which was the outside. The factory confirmed that the feet coming out of the injection molds will have a tooling mark on one side, making it much easier for the assembly line workers to do the right thing.

And then we found the big problem with the sample. When the keyboard is "negative tilted" with the front higher in the air than the back, it's dramatically unstable. The front foot acts like a fulcrum and the keyboard pops up off the desk when you rest your palm on the palm rest. Now, there's an argument that you should never actually rest your palms on a palmrest, but that argument is kind of bullshit.

We didn't catch this issue when we were testing samples in Shenzhen because we didn't yet have an integrated sample with the wood top and the feet fitting well enough that we expected it all to work right (or we would have brought it home with us.) We've raised the issue with the factory, they've replicated it and are thinking through potential solutions (which we're pretty sure will involve moving the front feet much, much closer to the front edge of the keyboard. We've also got a few ideas of our own if they don't come up with something acceptable. If nothing else works out, we may end up being really, really happy we decided to add 1/4-20 tripod mounts as attachment points on the bottom of the keyboard.

There's also a chance that with the electronics installed, the change in center of mass will mean the issue will be significantly better, but we're not holding our breath.

We had a call with the factory late Sunday night and ran through all of these issues, as well as a few other minor items. They're going to build out a fully integrated sample this week, test it locally with whatever improvements for the feet they can come up with and send it over to us.

On a happier note, the slide-on center bars seem to do what they're supposed to and support the keyboard in both flat and tented configurations.

Electrical

While we were waiting for the mechanical sample, the factory sent us sample PCBs to test. Those showed up on August 23.

The factory sent four sets of PCBS. Two of them had the APA102C LEDs we'd specified and two of them had "compatible" HOD9822 LEDs from Haodong, a Shenzhen brand. (You might note the quotation marks around the word compatible in the last sentence. We'll get to that in a moment.)

The factory proposed the Haodong LEDs as an alternative because the APA102C LEDs need to be soldered at a relatively cool 230°C, rather than the 260°C they usually use for their reflow ovens. If you solder the APA LEDs at 260°C, they have a tendency to… cook inside and never glow again. The Haodong LEDs are somewhat hardier and can handle the higher temperatures without breaking a sweat. The Haodong LEDs speak the same wire protocol as the APA102Cs. The Haodong LEDs are also just a bit cheaper. So far, so good.

In testing, the problem comes when you start driving them at high intensity. If the APA LEDs are a little bit starved for power, they get dimmer. If the Haodong LEDs are a bit starved for power, their color changes a bit. If the APA LEDs are really starved for power, they get dimmer. If the Haodong LEDs are really starved for power, they start to behave somewhat erratically.

So, as of now, we're saying no to the cheaper, easier to solder LEDs.

Since last month, we’ve also been tracking one other issue with the PCBs. We’re using PCA9614 controllers to talk across the cable between the two hands, via the I2C protocol. They work great. However, when we were debugging an unrelated using a FLIR heat camera, we discovered that the PCA9614 chips were heating up to about 50°C. They're not ordinarily supposed to get that hot, so we reached out to NXP, who make the chip. Our NXP sales person connected us with a field applications engineer (FAE).

FAEs are essentially the most technical of technical support. The NXP FAE vetted our schematics and, after building a clone of the relevant circuit on a breadboard, agreed that the design looked ok. The next step was to evaluate an actual set of boards, so we drove down to San Jose with a set of boards and walked through the issue with our FAE. He still didn't see anything wrong with our design, so he asked if he could borrow the boards to test in their lab. As of last Friday, he'd confirmed that the design of the boards looks ok and that they're seeing the heat issue with our boards in their lab. They say that this week, they've got someone on the chip design staff looking at the issue.

According to the datasheet, the PCA9614 chip’s temperature is within its specced operational range, but it makes us uncomfortable that it’s getting so warm. Our worry is that if the chip is giving off this much heat, it might be slowly roasting itself. This might ultimately turn out to be nothing, but we're not sure it's nothing, so we'd rather have NXP check it out while we can still fix it.

Other than that, the electronics seem to be behaving themselves. We noticed that we'd accidentally printed some text on one of the boards backwards. So we'll get that fixed up before production.

Firmware

Nearly all modern electronics have firmware, essentially programs that control how they behave. These days, most of that firmware is programmed into devices before they leave the factory, never to be updated or improved.

Sometimes, that firmware is programmed/flashed onto devices after they've been partially assembled. Sometimes, it's flashed directly onto the device's microcontroller chips by a chip manufacturer or distributor before they even arrive at the factory. There are a couple reasons you might do this. The first one is that your firmware might be a trade secret and you might not want have your assembly factory to be able to make your product except with the exact chips you send them. Needless to say, this doesn't apply to us. The second reason you might send pre-flashed chips to your factory is that it can be faster and cheaper to program them in advance.

Our factory would strongly prefer that our chips get flashed by the distributor before assembly. For us, the only real downside to this is an earlier and firmer deadline for the shipping version of the firmware. Unlike most manufacturers, we _want_ you to be able to reflash your keyboard (to configure custom layouts, build neat add-ons, change the LED effects, etc). That said, we want to make sure you can't accidentally "brick" your keyboard by flashing buggy firmware onto it. Luckily, chip designers are well aware of this kind of issue and field-reprogrammable chips often have space on them for two completely different programs: the firmware and the "bootloader." The bootloader is responsible for either running the main firmware or letting you replace the main program with a new version. We've been focusing our ATMega32U4 programming efforts on making sure that our bootloader is robust and reliable. The keyboard will, of course, ship with working firmware, but by the time it reaches your hands, there will almost certainly be better-working, more amazing firmware available to update to.

Early on, we decided to make the Model 01 "Arduino-compatible", so we chose to base our bootloader on the same "Caterina" bootloader used by the Arduino Leonardo. Caterina is designed to make it relatively foolproof to update the firmware on an Arduino over USB without having to tap a reset button on the device or connect a specialized programmer to your board. The only thing Caterina won't let you do is replace your bootloader with a new one.

For a development board like an Arduino, this is a reasonable compromise between security and convenience. For your keyboard, it's another story. While we know it's not terribly likely, it's important to us that malware won't have an easy time reprogramming your keyboard. When we designed the Model 01, we connected a pair of pins on the ATMega32U4 that's your keyboard's brain to the top left key on your keyboard. We've modified the Caterina bootloader not to go into programming mode when it's reset by software unless you're holding down that key. We've also updated the bootloader to be able to use the keyboard's LEDs to tell you what's going on.

You can find source code for our version of Caterina here: https://github.com/keyboardio/Model01Bootloader

When you want to update the firmware on your Model 01, you'll be able to use the free Arduino IDE, the avrdude command-line tool or a little custom GUI tool we're building. If you decide you want to replace the bootloader on your Model 01, you'll need to unscrew your keyboard and connect a hardware programmer (or an Arduino pretending to be a hardware programmer) to the ATMega32U4's "ICSP" header.

While the ATMega32U4 is the Model 01's brains, another pair of Atmel chips act as its, uh, nervous system. A pair of ATTiny88 microcontrollers are responsible for scanning the keyboard's keys and telling the LEDs what color to turn. They talk to the ATMega using the I2C protocol. The ATTiny chips don't have a special protected part of memory where a bootloader is supposed to live. Typically, they're used in embedded applications where you're tight on space and probably won't be updating their firmware often, if at all. Because of this, we've been working hard to make the firmware on the ATTinies robust and configurable. Our initial firmware hardcoded some sane defaults that we think will work well, but after a bit of reflection, we went back and added some extra features to the wire protocol that let us tweak the ATTinys' behavior if we later decide that our sane defaults weren't actually so sane.

You can find the ATTiny firmware here: https://github.com/keyboardio/avr_keyscanner

The ATTiny88 may not have specialized support for a bootloader in protected memory, but it does have the ability to reprogram itself. Last week, we discovered that many years ago, Atmel published a technote (AVR112) on how to build an I2C bootloader for devices that don't have a protected bootloader memory area. This week, we've been working on a derivative of the AVR112 bootloader that, if we pull it off in time, will let you update the ATTiny firmware just by flashing a special programming tool onto your keyboard. The code is still...a work in progress, though you can find it here: https://github.com/keyboardio/attiny_i2c_bootloader

This is definitely the sort of fiddly embedded code that would most benefit from advice, help and criticism if you're so inclined.

The actual initial keyboard firmware has seen only the tiniest of tweaks in the past month. You can still find it at https://github.com/keyboardio/KeyboardioFirmware

We've also started work on Arduino IDE hardware support for the Model 01. This is, essentially, a bundle of a few config files, all of the Keyboardio-specific Arduino libraries, a hardware definition, our bootloader and build instructions for the Model 01's firmware. By installing this in the Arduino IDE, you'll be able to build the Keyboardio firmware for the Model 01 and flash it just like any other Arduino sketch.

It's very much a work in progress, but "Works On My Machine"[TM]. Right now, you install it by checking it out of git and syncing all the submodules. When it's fit for use, we'll publish tarballs with everything already checked out of the subrepos and built. We'll also publish it in a format installable through the Arduino IDE's board manager.

You can find the current version here: https://github.com/keyboardio/Arduino-Boards

Keycaps

As reported in the last backer update, we finalized the keyshapes before Jesse left Shenzhen. We're still waiting on final pricing and MOQ for alternate keysets, but it's time to get a sense of what you want on your keycaps. No matter what you want on your keys, please fill out the non-binding survey here: https://keyboardio.typeform.com/to/Zh86UM

That'll help us get a sense of whether we can pull off runs of Colemak, Dvorak, AZERTY, QWERTZ, Runic, etc keycaps.

Schedule updates

The factory owed us an updated schedule spreadsheet this week, but it hasn’t shown up yet and we don’t want to hold this update any longer.

As of about 10 days ago, the first “T0” shots from the injection molds should start showing up next week. We’re still not sure exactly what’s going to happen with the reverse-tilt issue with the current design for the feet, but it sounds like a future update will talk about how one modifies an injection mold that’s not-quite-right. Our rough understanding is that it involves welding steel into the molds before remachining them.

We heard from Matias yesterday that the first 10% of our keyswitches will be delivered by the end of September, with the rest to follow soon afterward as they spin up their production lines.

Jesse’s headed back to Shenzhen for a few weeks on September 18, just after the Chinese mid-autumn festival. He’ll be working with the factory to resolve the issues we’ve talked about in this update and to check over the output of the injection molds.

There’s still plenty of work to do, but we’re getting ever closer to mass-producing your keyboards. As always, we’ll keep you updated on Twitter, in the Kickstarter comments and with future backer updates.

<3 j+k

Day 420: Hinges and feet and keycaps and PCBs
over 7 years ago – Wed, Sep 07, 2016 at 04:41:09 AM

TL;DR: Jesse extended his trip to Shenzhen by a week. We’ve solved many of the issues mentioned in the last update. The first samples made by the factory weren’t quite right. Second samples should be done in the next week.

Hello from Shenzhen and Oakland,

Things are proceeding apace, though it hasn't all been sunshine and puppy dogs.

The design process took some time. Jesse's been working directly with the factory's mechanical engineer, electrical engineer, head of R&D and our amazing sales person, Maggie. Most days at the factory start around 9:30 and wrap up somewhere between 8pm and 10pm. For the past two weeks, we've all been gathered in a conference room, working through design and assembling prototypes. One afternoon, the power went out, which meant no AC and, eventually, no light. So we ended up designing by mobile phone light.

Mechanical

On the mechanical front, we've made amazing progress over the past few weeks.

We've got feet that work. We've got an incredibly simple interconnect mechanism that, working with the feet, will let you tilt and tent your Model 01 at the same time. We've got a 1/4-20 tripod mount on the bottom of each half of the keyboard, so that you can do crazy things with your Model 01's placement and angle if you want to.

Getting to the first samples from the factory took a bit more time than expected.

First, we went through a couple interconnect mechanism proposals. Nothing was really working all that right, so we stepped back and looked at the sliding dovetail mechanism Jesse and Brian Garvey designed for the Highway 1 Demo Day prototype. We were in love with the design, but a mechanical engineer we were working with talked us out of it, since he said it'd be way too expensive to make.

Flash forward to the point where we're working with a factory, who can actually tell us what it costs to make various solutions. The sliding dovetail turns out to be pretty workable.

We got first samples of the new enclosure and sliding dovetails back late last week. (They were actually supposed to be back last Monday, but the factory's prototype supplier completely blew their deadline. And their second deadline. And their third deadline. And then the parts they got back to us had a bunch of machining errors. Thankfully, we were able to use the bad parts to learn a lot about the updated design, including numerous things that worked great and a few things that just weren't right.)

For the previous prototype, we had a single bar you could flip over if you wanted to switch the keyboard from a flat configuration to a tented configuration. That only worked because the feet we were using at the time were bolted onto the bottom of the keyboard. We, of course, still plan to let you tilt and tent your Model 01. With the new design, the feet are actually built into the bottom plate, so the flat center bar needs to be a lot thinner.

The plan is to ship with a thin, flat center bar and a tented center bar with stabilizing feet built in for when you're using your keyboard tented and tilted on a desk.

The feet we designed were a collaboration between Jesse and the factory's mechanical engineer. They're absurdly simple. But they get us tilted and tented configurations when the keyboard is connected together or when the two halves are separated.

For a while, we thought we were going to need to do something clever and complex, like our friends at UGL did for the Ultimate Hacking Keyboard. They designed screw-on feet that users can rotate to get the different angles they needed. We looked at feet that locked at multiple angles. We looked at feet that had screw-out extensions. We actually made screw-on feet that locked at multiple angles and had screw-out extensions. They were expensive. They were complicated. They were a pain to set up. What they weren't was particularly sturdy.

What we ended up with is flip-out plastic feet, similar to what you'd see on most keyboards, but with a few key differences. Rather than being placed parallel to the front and back of the keyboard, they're placed at a 45 degree angle. Also, the bottoms of the feet aren't flat. They're cut with three different angles on them. Inside each foot is a mini-foot. With these feet, you can tilt your keyboard forwards or backwards. You can tent your keyboard so the middle is higher than the outside and and you can tilt and tent your keyboard at the same time. You can also reverse-tent your keyboard, so the middle is lower than the sides. But don't do that. It's bad for your hands.

The last time Jesse was in Shenzhen, he met with two wood suppliers. One was the supplier found by our ex-factory. The other was a new supplier he met through one of our candidate factories. In June, we got samples from that second supplier. They were really, really not very good. The supplier apologized and offered to make new samples. We visited them the week before last to pick up the updated samples and to discuss how the project might move forward if we picked them. One of the new, supposedly perfect, samples had a crack that had been glued back together. To say we were disappointed or frustrated would be something of an understatement.

We also got updated wood samples back from our preferred wood supplier, with whom our factory is now working directly. They look and feel fantastic. :)

Yesterday, we went through a comprehensive mechanical design review, so that the factory could place an order for new mechanical prototypes today. We're expecting the new samples by this weekend.

Keycaps

The keycaps have seen only minor changes. The Escape and Butterfly keys got slightly reworked to have a cleaner interior corner. The thumb keys got slightly tweaked so that they won't get caught against each other when pressed down. Also, all the keys all finally got numbers embossed on the insides. Initially, the plan was for the keys to get the same "row and column" numbers we use on the PCBs and in the firmware. The factory's team protested that they usually do things differently, just giving each key a number. We said that that was very nice, but that our way would be easier for users to figure out without a diagram. A couple days later, we finally came to understand that the numbering system the factory proposed was actually about making assembly easier and more reliable for the staff on the production line. Once we understood that, the decision was an easy one. The keys will be numbered from 1 to 64, so they're easy for the factory's assembly line staff to assemble. We'll provide you with a nice diagram showing where each key goes.

As of now, the only thing left on the keycaps is to verify that once they're painted and UV coated, everything still fits well. The test keycaps got painted on Monday and we should see them tomorrow.

It sounds like our minimum order quantity for alternate key legends (like Dvorak, Colemak, Bepo, Nordic, etc) is about 500 sets. We're still waiting on the costs, but at a guess, we'll be able to do alternate labels if we have 100-125 preorders for extra keysets of a given layout. The nice thing is that production of those keysets is something we can spin up pretty quickly once the initial injection molds are made. In the not too distant future, we'll run a survey to gauge interest in the likely alternate layouts.

Electrical

When last we wrote, the big blocker on the electrical front was availability of the ATMega32U4 chip the Model 01 depends on. We jumped through a bunch of hoops with two potential resellers and even reached out directly to Microchip, who now own Atmel. Things were looking complicated, expensive and slow. We started asking everybody we met if they had a secret source for the ATmega chips we needed. We debated just swallowing a couple thousand dollars in extra costs, buying them from DigiKey in the US and FedExing them to Shenzhen. And then we asked our friends at Arduboy, who make a really neat pocketable game console based around the ATMega32U4. As it turns out, the supplier they're getting their chips from can't really help us, but one of their other suppliers happens to have a whole passel of the chips we need sitting in their warehouse in Shenzhen. So we bought them. Yay!

Over the past week, we've been working with the factory's electrical engineer to triage why the sample PCBs they made for us just weren't working right.

As we've worked together, the sample boards have gone from "Wha???" to mostly, sorta, kinda working and then to “seem pretty ok”

“Seem pretty ok” isn't really quite good enough. We think we know what went wrong and we'll be pushing hard on the electrical front this week.

Some of the problems came down to issues with hand-soldering tiny components. Careful testing and rework have solved these issues. This won't be an issue in production. All of the boards will be assembled by machine.

Some of the problems may be related to parts substitution, which can be a really serious issue when getting electronics made. The way it's supposed to work is that a product designer will specify which components of an electrical design will be 'consigned' to the factory by the customer, which components will only be purchased from an 'assigned' manufacturer or distributor, and which parts are 'generic' and can be purchased from whomever the manufacturer wants. In our case, there are no consigned components, but a number of assigned components. We're particularly picky about the Atmel microcontrollers and the APA LEDs, but we also care deeply about the RJ45 jacks (specified as coming from TE), the USB Type C jack (from JAE), and a bunch of other, more minor stuff.

The factory agrees 100% with us that under no circumstances should they be substituting any parts without our express authorization. But, when building the prototype boards, that's just what happened. The two places it was most visible were in the USB Type C jack, which didn't latch very well and in the RJ45 jack, which had a slightly different PCB footprint than the one we'd specified. As we dug into the issue, what happened was that some of the suppliers weren't able to get samples of the parts we needed on short notice and just substituted parts they thought would be a good match. That's not going to happen again.

As the factory researched what had happened, a couple of their suppliers expressed frustration that we had specced expensive foreign parts, when there are solid locally made alternatives that they believe will work just as well. The factory's proposal is that for the next test build, they'll make two versions of each board. One with local versions of parts and one with the versions we'd originally specified. We'll get a copy of the proposed alternative BOM and part numbers and datasheets for everything substituted and then we can make an informed choice about BOM changes. They've already set us up with new options for the RJ45 and Type C jacks that are totally reasonable. As we make our choices on component substitution, we'll be focusing on quality, availability and support, more than on price.

(A brief note on the USB Type C jack: we’re shipping with a USB A to C cable. So the bit that plugs into the keyboard is the small and reversible Type C, but the other end will fit into the Type A USB port that's already on your laptop, desktop, or USB hub.)

By far, the biggest problems were, we think, related to a miscommunication around assembly instructions. When we designed the two PCBs for the two halves of the Model 01, we designed them as separate boards, rather than one board with two halves. As such, we repeated certain part identifiers between the two boards. For example, both boards had a resistor numbered "R14". On the left hand, R14 is a 10k Ohm resistor. On the right hand, R14 is a 500k Ohm resistor. Raise your hand if you can see where this is going. So yeah, the EE didn't work from the schematics or parts placement files when assembling the boards. Instead, he worked from the combined BOM we'd sent along with the bid packet for pricing. This was the result of a misunderstanding, the language barrier, and a bad design choice on our part. The misunderstanding has been corrected and the part numbers on the right hand board have all been updated.

The last ‘big’ problem, once we got the components on all the boards sorted out, was that the tool we use to flash the firmware onto the ATTiny chips that drive the keyscanning engine doesn’t work quite like Jesse thinks it does. We spent two days chasing our tails because the boards worked erratically. The factory brought in a partner company to help triage the boards. While they were working on one set of boards, Jesse went across town to do some debugging with Charles Pax, an American hardware hacker living in Shenzhen. Charles makes a nifty temperature sensor and data logger called the Pax Instruments T400. (Yes, that is a plug for his product. He didn’t ask for it. But he deserves it, because he’s a hero.) 

Charles happens to have a very reasonable EE bench and rework station at his apartment in Shenzhen. Working with Jesse, he traced the issue down to a difference in the behavior of the ATTiny chips on the boards we brought from the US and the ATTiny chips on the local boards. Charles swapped the chip on a ‘good’ board and a ‘bad’ board and magically the bad board started to work. From there, a little bit of investigation pointed the finger at the ‘fuses’ on the bad ATTiny. The fuses are, essentially, how you configure the chip. And you need to actually set them correctly or your chip will work erratically or not at all.

Now that we know what went wrong, the most astonishing thing about the first sample circuit boards is that they work at all. Seriously. It's kind of nuts.

Over the weekend, the factory did a test of automated assembly of the PCBs. Aside from some issues with overcooking the LEDs the first time through the solder reflow oven, the new boards work much, much better.

What's next

Jesse's headed home to Oakland tomorrow night. Right now, it’s looking like he won’t be heading home with a fully integrated sample unit of the Model 01. We’re expecting it in about a week by FedEx or DHL. Once we have the sample, we'll be able to formally sign off on the design and the factory will start work on injection mold tooling.

The Schedule

The delays on getting the prototype’s mechanical parts built has put us 1-2 weeks behind the factory's most ambitious estimated schedule. (That is, 1-2 weeks past October 19.) They owe us an updated schedule and we expect it later this week, but didn't want to hold the backer update for it.

<3

j + k