Day 827: A surprise trip to Shenzhen to QC the first 1000 MP keyboards
over 7 years ago – Wed, Sep 20, 2017 at 12:35:32 AM

TL;DR: QC of the first 1000 keyboards starts tomorrow. They should be on a plane to the US by Monday; The wood factory is working on the second 1000 enclosures, but hasn’t promised a delivery date yet; Jesse’s in China for an unplanned trip to do QC; Bay Area meetup November 11th.

Hello from Oakland and Shenzhen,

(Right now, Jesse’s on the ground in Shenzhen for an unexpected factory visit. More on that later.)

When we wrote last month, we’d been pretty certain that keyboards for all Kickstarter backers would be on their way by now. That did not come to pass.

As of today, the factory has completed assembly of the first 1000 Model 01 keyboards. They’re currently going through a 72 hour burn-in test.

Why only 1000? Our wood factory was having trouble meeting their deliverables schedule, so we decided to split the Kickstarter manufacturing into two batches of 1000 units. The hope is to deliver as many high-quality Model 01s as we can, as soon as we can, even if that means they won't all be delivered at the same time.

Overall, the latest delay has primarily been due to quality issues with the wooden enclosures. As the factory checked the enclosures that had arrived from the wood CNC shop, they found a number of defects.

These defects ranged from discolored "spots", a few cracked pieces, mis-installed brass screw nuts, and several hundred pieces where the wood seemed to have been over-milled such that there was a big gap between the baseplate and the enclosure.

The wood factory is taking responsibility for the issues and has been working overtime to rework and replace the bad enclosures.

Obviously, we're pretty frustrated at the slowdown in production, but in the end we recognize that we'd much rather have caught these issues now rather than shipping out bad keyboards.

Now that the wood factory has delivered 1000 good enclosures, they’re working on the next 1000. After that, they’ll deliver another batch. Once those keyboards are built, we’ll be caught up on preordered keyboards.

There are a bunch of factors that will influence the order in which we send out keyboards, but one important one is the BackerKit survey. If you have not yet filled out the survey confirming your address, you should do that. If you didn’t get the survey, please write us at [email protected].

Inspection

Typically, the way pre-shipping quality inspection for a consumer item works is that a third-party inspector does a spot check of a small number of randomly selected units from the upcoming shipment. (This is in addition to controls the factory has implemented internally.) If no serious issues are found, the shipment goes out the door.

We were not terribly happy with some of the quality issues that made it out the door on PVT keyboards. Since then, we’ve been working with the factory to help them figure out process improvements to reduce or eliminate the kinds of issues we saw. We've also been beefing up our documentation of the quality standard and test procedures.

The factory does significant testing as part of the assembly process. At the same time, we decided that we’re going to do a relatively expensive "100% check" of at least the first few hundred units in the first mass production run. Based on the results of that check, we’ll decided whether to continue with the 100% check or back down to random sampling.

When we contacted our third party QC firm a week ago to schedule their inspection of the first 1000 keyboards, they told us that they might have one day for us next weekend, but are otherwise unavailable. They also told us that they won’t be available for most of October.

That was... unexpected and somewhat disappointing.

Since then, we’ve been scrambling to find a replacement quality inspection firm. If the Model 01 were a ‘regular’ keyboard, we’d be comfortable letting a QC firm use their standard quality check procedures without any special training. But the Model 01 is not a regular keyboard.

Since the quality inspectors Jesse trained to check the Model 01 aren’t around, we didn’t see any other option than to put Jesse on a plane to Shenzhen. On Wednesday Jesse will start the quality inspection, along with a freelance quality inspector recommended by a friend of ours as a sort of trial run. At the same time, we’re continuing to talk to a few other options to see if one of them might be a better fit.

Keycap sets

While in Shenzhen, Jesse should be approving the plastic packaging for the ‘extra’ keycap sets. If the packaging looks right, the tooling to produce it will take about two weeks. There’s a decent chance that the extra keycap sets will ship with some keyboards, but we’re pretty sure that we’re going to end up shipping some Model 01s to customers and then shipping your extra keycap sets as a followup as soon as they’re available.

Box of crap

If things go well with the factory, Jesse may have a few days in Shenzhen to do another run of our “box of crap from Shenzhen” project. If you think you might want a box of crap from Shenzhen, keep an eye on http://twitter.com/keyboardio. We may try shipping everything to ourselves in California and doing fulfillment from the US time around.

Bay Area Meetup

We're taking part in the big Bay Area Mechanical Keyboard Meetup that's happening on November 11th in Palo Alto. There will be a lot of keyboards there, ours and others' and old rare ones, plus some talks and good times. The event is free, but requires advanced sign-up. Hope to see some of you there!

<3 j+k

Day 795: Reports from the field
over 7 years ago – Fri, Aug 18, 2017 at 01:16:07 AM

TL;DR: PVT keyboards are in customers hands; we've been working to ensure that issues found during the PVT run will be resolved before mass production starts next week; we're working with BackerKit to get surveys out to everyone who preordered a keyboard from our website.

Hello from Oakland,

Since we last wrote in June, we've shipped the first 65 keyboards to Kickstarter backers who offered to help us test keyboards from the pilot run. At the same time, we've been working with the factory to get ready for Mass Production.

 

 

Mass Production starts next week

Mass Production is currently slated to start on August 25. The factory believe it will take them about a week to make the 2000 units to cover our Kickstarter preorders. After that, the keyboards will be tested in Shenzhen and be packed for their journey to our warehouse in California. Last time around, that took a little more than a week.

Once they arrive, we'll spend a day or two doing additional spot checks to make sure everything looks right, and then we'll ask the warehouse to start shipping them out as fast as they can.

On September 21, the factory will start "MP2"—the second mass production run. That will cover all post-Kickstarter preorders, as well as a whole bunch more. Right now, they're estimating that MP2 will be done by October 7, though we're looking at options for shipping them from China in blocks of 500 as they're finished and tested. 

We've started working with BackerKit to get preorders from our website into their system, so we can send out surveys to everyone who's preordered. If you've preordered from our website, you can expect email from BackerKit as soon as the survey is ready. We'll also tweet about it. 

PVT Keyboards

Generally, the response to the PVT keyboards has been fantastic. We tried shipping the keyboards without "overpacking" them into larger cardboard boxes around the boxes. Initially, we were a little bit worried that the box might not offer enough protection, so we asked customers to tweet us some pictures of how things looked when they arrived. The packaging did its job—every keyboard arrived in good condition.

 

The point of a PVT run is to do a limited production run so you can fine-tune the process and figure out what sorts of problems you're going to have. Using that information, you can update your assembly and test procedures and figure out if there are any design or materials issues that need to be addressed.

As expected, the keyboards haven't been without their issues.

The biggest assembly/test issues we've run into are: soldering issues, wood surface issues, and keycap printing issues.

Soldering

We knew that there were a number of soldering problems with the PVT units, mostly due to the outsourced PCB Assembly factory they used to hit the PVT deadline. Even before we'd shipped, we'd identified some potential issues and they had done a first pass of additional quality checks. On a few keyboards, there were issues they'd missed, mostly related to bad solder joints. While PVT backers have been very understanding, this is something that can't happen for mass production and the factory has told me that they've switched PCBA back to their standard supplier for mass production. At the same time, they've been working on additional manual QC steps to ensure soldering problems don’t slip through to shipped units for MP.

Wood surface

Some PVT users have found that the right hand enclosures for their keyboards have a slightly rougher surface than the left hand enclosure. We discovered this issue when we were checking the wooden enclosures before we shipped, but decided that it wasn't a critical issue for PVT keyboards. We've told PVT customers who find it distracting that we'd be happy to swap out their enclosures once MP units are available.  The reason this happened is that the wood factory made all of the left hand enclosures for PVT on one day and all of the right hand enclosures for PVT the next day. We've worked with the factory to document a "quality standard" for the wooden enclosures, using problems we found during PVT as examples. As the wood factory delivers batches of new enclosures to our factory, they're doing a second check of them and sending units that don't meet our requirements back to the vendor for rework.

Keycap printing

We've written before about the quality issues we've had with keycap laser engraving. So far, two PVT backers have reported keys swapped with each other, one backer has reported a key that had been lasered upside-down and one backer had a second 'J' key where his 'L' should be. We swapped out the affected keys. Most of the keycap engraving issues were the result of a crazy manual process the factory was using to build sets of “good” keycaps from imperfect sets they got back from the laser engraving supplier. The factory has been working hard to improve the laser-engraving process. They've upgraded the painting jigs to make sure that the keycap alignment stays consistent on the laser engraver. They've built a custom tool to help make sure that the right keys get placed in the right positions for laser engraving and they've spent time training the technicians at the laser engraving factory to make sure that the keys are correctly placed on the jigs. We've also worked with them to clean up the placement of the legends a bit and slightly decreased the font size of all the key legends with words like "shift", "num", "return", "alt", etc. The factory has also been working with the keycap injection molding factory to correct some small errors in the keycap tooling.

Tight RJ45 cables

One issue some folks have run into is that short RJ45 cable require a lot of pressure to "lock" into the jacks on either half of the keyboard the first few times you do it. Other than the tight fit, the cables work fine. This is, we believed, caused by a tolerance issue between the jacks on the keyboard and the specific style of connector on the cable. The cable supplier has been looking for alternate connectors, but as yet, everything they've tried fits similarly. For the first mass production run, we'll be sticking with these cables. If the factory manages to find a better option, we might change them for future runs.

Arrows of Ambiguity

There is one design issue that we're working with with the factory to correct. The sliding dovetail bar and rail system used to connect the two halves of the keyboard has been confusing to many users. The rails are tapered to ensure that the center bars slide on easily and click into place when slid in the correct direction. We printed arrows on the rails on the bottom of the keyboard to help show which direction to slide the keyboard into the center bars. Unfortunately, the arrows turn out to be confusing for some folks and it's possible to force the bars far enough onto the rails to get stuck if you slide them in the wrong direction. Once you understand how the center bars are supposed to be connected, it's pretty easy to keep doing it the right way.

 

We've updated the manual to better explain how to orient the center bars and have made a small design change to the rails that makes it impossible to slide the center bars on in the wrong direction. The factory is working hard to modify the injection molding tooling for the rails, but there's a chance it won't be completed before mass production for Kickstarter keyboards is done. If that's the case, we'll be providing Kickstarter backers with an additional reminder about the right way to connect the center bars as we ship their keyboards.

Software

Our PVT users have reported a number of issues with the firmware shipped with PVT keyboards. The big ones were that Media Keys and mouse button emulation didn't work right on macOS. There were a few additional bugs reported around key repeat and LED modes. We believe we've resolved all of the above issues in the Mass Production firmware, which we had to send to the factory on Sunday. Of course, one of the things about the Model 01 is that we plan to continue to improve your keyboard's firmware for a long time to come.

We also made a change to the keyboard's "bootloader", which is the program that's responsible for letting you update the keyboard's firmware. A number of PVT users have made changes to their keyboard's firmware ranging from changing the key layout to writing cool new LED effects. A few of them have managed to 'brick' their keyboards, making it impossible to get into the bootloader to reprogram their keyboards without unscrewing the enclosure and hitting the handy 'reset' button we've put on the PCB. Because of this, we've added a new feature to the bootloader: If you hold down the 'prog' key when you first connect your keyboard to your computer, it will enter the bootloader without trying to execute your possibly-broken firmware and you'll have a little bit of time to flash a new program.

 

Along with all of this, we’ve been working to clean up the firmware source code, to fix a number of portability issues in the build tool, and to improve the code’s documentation. We've also started to write user-facing documentation. We're tracking the documentation tasks here: https://github.com/keyboardio/Kaleidoscope/projects/1

There are a few 'major' features which aren't in the shipping keyboard firmware, because we aren't confident they're safe and stable enough. The two biggies are iOS 10 support and "Boot Protocol" keyboard support. We will provide both features as part of a free firmware update. (All firmware updates for the Model 01 will be free.) 

• Apple appears to have made a change to how iOS 10 handles USB devices that say they want more than 100mW of power. In earlier versions, iOS would warn the user that the device wasn't supported, but the keyboard would still work. Now, iOS refuses to let the device connect. We have a version of the firmware that works just fine (but without LEDs) on iOS, but need to do some engineering to make sure that we aren't committing horrible atrocities against the USB stack on every other device to support iOS.
• The USB specification defines, essentially, two kinds of keyboards. Regular keyboards and "Boot Protocol" keyboards. Boot Protocol keyboards speak a simplified version of the USB keyboard protocol designed to be a little easier to implement in a computer's BIOS or pre-boot environment. The downside of the Boot Protocol is that it doesn't support "NKRO"—you can only press six keys at the same time. Our core libraries speak both keyboard protocols, but Kaleidoscope doesn't yet do the right thing when the host wants to talk Boot Keyboard rather than NKRO keyboard. We ended up deciding that the changes required to make the keyboard do the right thing were too potentially destabilizing to add just before locking the firmware down for mass production. Getting this fixed is a high-priority task for Jesse, and we're hoping to have it available as an update in the near future.

As always, our firmware lives at https://github.com/keyboardio/Kaleidoscope/ and new contributors are most welcome!

 

Builderscon 2017

Earlier this month, Jesse gave a talk at Builderscon about our experiences finding and working with a factory. The conference is primarily intended for the local audience and his talk was one of the few in English, though the conference organized simultaneous translation so folks actually laughed at some of his jokes.

All in all, the conference was really well organized and folks seemed pretty excited about the Model 01. One of the neatest talks Jesse saw, but didn't really understand, was the story of one intrepid hacker's adventure designing 3D models for a difference engine and 3D printing all the parts.

There's already a video of Jesse's talk up on Youtube. It's about an hour long and is a reasonable (if slightly simplified) summary of the last two years of our backer updates.

 

Tokyo Meetup

The day after the conference, we held a little meetup at FabCafe in Shibuya, Tokyo. It all came together at the last minute and while we tried to email backers we knew were near Tokyo, we know it was short notice.  It was lots of fun. FabCafe is a super-neat space. Their primary space is a trendy coffeebar with laser cutters. Hidden away upstairs is a proper makerspace with bigger tools and meeting space. As we got too big for the cafe, they very graciously offered us up the private space upstairs.  About fifteen people came by to check out the Model 01 and to chat keyboards.

 

Boston Meetup?

We're going to be in Boston for a family event from August 28-31 and are trying to figure out if we have the free time to host a little Keyboardio meetup. If we do, we'll announce it over on https://twitter.com/keyboardio. (Alternatively, email [email protected] with subject line "Boston meetup" and we'll make sure you get an email with details.)

<3 j+k

Day 764: BackerKit surveys are coming
over 7 years ago – Tue, Jul 18, 2017 at 12:31:22 AM

Hello from Pasadena, CA!

In the next couple days, you’ll receive an email from us with a special link to your BackerKit survey. It'd be very helpful for us if you can respond to your survey as quickly as you can since we need this information to fulfill your rewards.

About 100 of you will get that survey...now. BackerKit has us send the survey to a subset of backers, just to make sure we haven't bungled things. Once we're sure that the survey looks ok, we'll push the big red button and send the survey to everybody.

You don’t need to create a BackerKit account to fill out your survey. When you receive the email with the survey, click the survey link to respond. Answer the questions about your reward preferences, provide shipping information, and buy some more keyboards and keycaps if you want to. 

After you respond to your survey, you can change your responses at any time before we close the surveys in a few weeks to ship your keyboards out.

If you've backed for a single quiet-click keyboard and fill out your survey within the first day or two after you get the email, you should see an option to receive one of 66 "PVT" keyboards from the pilot run. These 66 keyboards are sitting in our warehouse in Gilroy, CA and will ship this week.

If you need to review your information or pledge status, you can return to your survey by clicking the link in your survey email or requesting your survey link under "Lost your survey?" on our BackerKit project page at https://keyboardio.backerkit.com.

If you used your Facebook account to log in to Kickstarter, the BackerKit survey is sent to the email address you use for your Facebook account. If you'd prefer to use another email address, please contact BackerKit support at https://keyboardio.backerkit.com/faq.

<3 j+k

Day 743: PVT
over 7 years ago – Tue, Jun 27, 2017 at 12:12:07 AM

Hello from Oakland!

TL;DR: PVT assembly is done; Mass Production should start next month.

We have quite a bit of news to report this month. Probably the biggest and most exciting news is that we're (finally) no longer three weeks away from starting the pilot run (also known as PVT). Roughly 85 PVT units were produced last week and are finishing up 72 hours of 'burn-in' testing now. On Wednesday, our third-party QC agency will double-check them as they go into their boxes. On Thursday or Friday, an express shipper should pick them up and put them on a plane destined for our fulfillment partner in California.

From there, they'll be dispatched to some early Kickstarter backers.

(We know we haven't sent your surveys yet. We're working on the survey design with BackerKit now. Look for email about the survey in your inbox in the next week or so.)

Long time, no backer update

It's been about six weeks since we last wrote. The first three weeks were pretty quiet, but the last three weeks have been mind-bendingly busy. The past week was really, really busy. We live-tweeted the PVT run.

Algernon collected our tweets from last week into a series of moments:

• Day one: https://twitter.com/i/moments/876743212114825216
• Day two: https://twitter.com/i/moments/877030769008533504
• Day three: https://twitter.com/i/moments/877393482255564800
• Day four: https://twitter.com/i/moments/877758532044685312

Jesse just returned from two weeks in Shenzhen. The plan was for him to spend a little over a week there finalizing the keycap laser engraving, checking out last minute details and supervising PVT on Monday, June 19. Things didn't go exactly to plan, but as you'll read, things went!

Keycap factory

Jesse's first day in Shenzhen, he visited our keycap factory to check in on updates to our injection molds and to talk through some issues with the trays they made for the laser engraving factory to use during the painting and engraving process.

They promised to get things fixed up right away and to produce sample keycaps within a day or two. When the caps still hadn't shown up by Friday morning, we got a little antsy and asked what the problem was. Our factory's team said they weren't quite sure, since the keycap factory claimed to have already completed the modifications.

At this point, we were prepared for any number of possible explanations or excuses. We've had a lot go wrong during the manufacturing process and thought that we were beyond being surprised.

The factory called up the keycap supplier, who told them that the modifications weren't done… and that getting them done that day might be difficult.

Because the factory had caught fire.

They reassured us that nobody had been hurt (and that our tooling was fine) and seemed to be up and running by the next day.

They did manage to get 120 sets of keycaps and painting jigs done in time for PVT, though the painting jigs appeared to still be of the older design that was prone to warping. Our factory doesn't know why the keycap factory hasn't delivered the newer painting jigs, but has promised to get to the bottom of it.

Keycap laser engraving factory

On his second day in Shenzhen, Jesse and the team from the factory braved typhoon Merbok to pay a visit to the keycap laser engraving factory in Dongguan. The primary issue on the agenda was figuring out why the keycap labels kept ending up significantly offset from where we'd put them.

We'd been sending files to the factory that showed the key labels with a variety of annotations to help the laser operator place them correctly. We'd gone so far as to build a layout that matched the painting jig to the millimeter.

Once we got to the keycap factory, the laser company took the sample keycap sets we'd brought with us and painted them black. They put them in the drying oven and suggested we go for lunch.

When we got back, we watched as the laser engraving technician printed out a copy of our alignment template, deleted all of the guide lines from the file, imported the legends into the laser engraving software, and proceeded to try to eyeball the correct label placement.

Going in, we were annoyed at how far off the legends were. In retrospect, it's astonishing how good his manual placement was.

After a bit of back and forth, we found a workflow that seemed to work better. We convinced him to keep the keycap top edge outlines in the drawing and to use those to make sure he was placing the labels correctly. That seemed to work better.

PCB Fabrication

The factory's regular PCB fabrication company wasn't able to make the boards in time for PVT, so they ended up using another supplier we recommended. These are folks that friends of ours have used on numerous occasions. Instead of the 20 days the factory's regular supplier quoted, these folks quote 6-7 day lead time. Well, they usually quote 6-7 day lead time. This time around, they quoted a 10-12 day lead time. Twelve days would have had the boards done on the Thursday before PVT began. But on Thursday, they told our factory that the earliest they could promise delivery was Monday. When the factory pushed back, they said that if we didn't like the lead time, we could try another supplier.

The factory escalated to a customer service manager who said that they could do Sunday, but that was the best we'd get. Sunday delivery meant that PCBA wouldn't start until Monday, which would delay PVT. This was not ideal, but we seemed stuck. Asking around a bit, some friends mentioned that the supplier had been "closed by the government for some reason."

We reached out to Ken Li, our amazing project management consultant. Ken said "What's our order number? Hang on a minute."

Ken came back about 10 minutes later saying that he'd reached out to the CEO of the PCB factory, whom he happens to know, and that the CEO had promised our PCBs would be delivered on Saturday. In the end, he was as good as his word and the boards arrived late on Saturday.

PCBA

The factory's regular PCBA (PCB Assembly) shop is located in the same building as the factory. Unfortunately, they don't work on Sundays and the factory couldn't get them to budge, so, in an effort to catch up to the promised PVT schedule, they reached out to another supplier who do work on Sundays. This was the first time they'd worked with this supplier. And, as of last Wednesday, it is the last time they will work with this supplier.

Every single PCB had problems. Most of the issues were mis-soldered diodes. The boards looked like they'd been assembled by hand, rather than by machine. They also looked like nobody had done a simple visual inspection after assembly.

Visual inspection is a standard part of commercial PCB assembly. Any halfway decent vendor has automated equipment to do visual checks of boards after assembly.

The factory ended up having to "rework" every board to fix surface mount assembly issues. This ended up being a two day setback. In hindsight, we would have been better off if they'd used their regular SMT shop, like they'll be doing for mass production.

Type C Cables

We thought we had a workable Type C cable for PVT, but when the factory placed the order with the supplier, they admitted that they didn't actually have the cables or the required raw materials to make more in stock and that we'd be looking at a lead time of 3 weeks or more.

Plan B was to wake up early on Saturday morning and go to Huaqiang Bei, the world's largest electronics market and buy some cables retail for PVT. Ken offered to come along and to bring a friend who was a professional cable maker who wanted a chance to pitch for our business.

Over coffee, he convinced us that we really, really didn't want to buy retail cables in the market and said he could build cables to our spec, on our deadline.

He ran off to find some samples of a few options. When he got back, we talked through our requirements and he promised to send samples.

Four days later, we had three sample cables that looked like we expected. Shortly after Jesse headed home from Shenzhen, the vendor delivered all the cables for PVT.

PVT production

Jesse didn't tell the factory it was his birthday, but they still delivered quite the birthday present. The factory fired up the assembly line and produced all of the PVT units on Wednesday, June 21.

We'll talk more about the assembly line in a future update.

QC

The next day, Jesse sat with the factory's QC team and our third-party QC agency to review the assembled units and to start to identify what quality issues they need to look out for and what we generally consider acceptable. For the sorts of parts they're used to dealing with, this is pretty easy. For some things, like the wooden enclosures, we needed to be a bit more detailed.

Wood is, of course, a natural material without a uniform color. We showed the factory examples of unacceptable problems, which included tooling marks on visible surfaces, wood filler on visible surfaces, and ugly discolorations. At the same time, they pointed out issues they thought we'd reject. For the most part, we agreed with them.

It was a pretty long day, but we managed to go through about 70 keyboards. The way it worked was that we’d choose a category of defect to review (like “keycaps” or “wood”) and then queue up each keyboard. The factory’s QC team would check the keyboard and either pass it or reject it. If they rejected it, they’d add a little arrow sticker pointing at the problem. If they passed it, they’d hand it to our QC guy. He’d review it, either adding a sticker and rejecting it or pass it and hand it to Jesse. He’d review the keyboard. If Jesse didn’t pass it, we’d stop the line, discuss the defect and the QC team would modify their acceptance criteria.

At the end of a given pass through the keyboards, we’d all review the rejected keyboards together to make sure they weren’t being overly critical.

The one surprise for everybody was "birdseye figure" Maple. Our jaws dropped when we found a few examples of birdseye maple in with the PVT keyboard enclosures. They were gorgeous. The factory's QC team thought they looked funny and didn't look like they were supposed to. It took a bit, but they now understand that if any birdseye Maple makes it into the supply chain, they shouldn't reject it. (Right now, we’re not able to promise anyone a birdseye figure enclosure. It’s very much luck of the draw.)

At the end of the night, the factory packed us up a carton of six keyboards, which Jesse hand-carried back to California, where we’ll be doing a little bit of additional QC.

Issues we found during PVT

By far, the biggest issues we ran into during PVT were with the quality of laser-engraving on the keycaps. At some point, we'll do a writeup of the root cause of the keycap issues, but that'll be a few thousand words on its own.

We ended up with an eye-watering defect rate of around 25% on the laser engraving for the keycap sets. The two biggest failures were poorly aligned legends and keycaps that had been laser-engraved backwards. The proximate cause of these problems was that the keycaps need to be moved (manually!) from one painting jig to another in order to make sure the bottom edges of the keycaps get good paint coverage. Because this process requires a human to touch every single keycap, it's a point where errors get introduced.

Needless to say, this isn't going to work for mass production.

We've spent a bunch of time with the factory going over process improvements for mass production. They want to do a bit of research and testing before deciding on a final solution, but we've got a few promising options.

What's next?

Now that PVT production is "done", there's still a lot to do before you get your keyboards. Thankfully, almost all of it is straightforward for the factory.

Once we get the PVT units to the states, we'll ship them out to willing early backers. (There will be a question on the backer survey about it.) After that, we'll be eagerly awaiting feedback from folks who get PVT keyboards to make sure that we’re not missing something obvious.

The factory will be subjecting a few Model 01s to torture tests. They'll be exposing them to a few days of ultra-bright UV, spraying them down with a saline fog, and dropping them on a concrete floor. While we sort of wish we could be there to watch these tests, we're probably better off not having to watch the torture taking place. That should take about two weeks.

The factory will also be sending one keyboard to the FCC test lab to send onward to their partner lab in Taiwan for final testing and issuance of the "Declaration of Conformity". That should also take about two weeks.

At the same time, the factory is working to refine their internal test and assembly process based on the PVT run. That'll let them build Model 01s faster and more reliably when they have to make a few thousand of them at a go.

Right now, we're expecting the first couple thousand units to roll off the line in July.

<3 j+k

Day 691: Crawling toward PVT
over 7 years ago – Mon, Jun 12, 2017 at 11:57:36 PM

TL;DR: Initial FCC/CE pretesting results; Component sourcing craziness; We're guessing that we're about two weeks out from starting PVT 

Hello from Oakland,

When we last wrote, we were pretty sure that April would see Jesse in Shenzhen overseeing manufacture of the first 100 Model 01 keyboards.

That didn't happen and we're pretty bummed out about it.

As of this writing, we believe Jesse will be getting on a plane to Shenzhen in two or three weeks. Below, we're going to lay out what's happened in the past month and what still needs to happen before we start manufacturing.

RJ45 connectors

When we wrote last month, we told you that the factory had enough RJ45 connectors for the first 1800 keyboards in their warehouse. While that is now true, it wasn't then. We only found this out about two and a half weeks ago, when the factory told us they hadn't yet sent keyboards to the test lab for FCC/CE pretesting.

As we dug into it, we found out that they hadn't sent the keyboards to the test lab because they couldn't finish building the circuit boards. They couldn't finish building the circuit boards because, while he factory's purchasing team thought they had received a large order of our RJ45 connectors, in fact the Chinese customs authorities had received them.

It took a while for the whole story to come out, but it sounds like what happened was that the customs declaration for the Taiwan-made RJ45 connectors wildly undervalued the product, so much so that Chinese customs took one look at the packages, said "Nope!", and impounded the goods.

We don't know why the customs declaration was so wrong, though we suspect that someone, somewhere was trying to be "helpful" to minimize taxes paid on the parts. We got somewhat cranky when we found all this out, and while we don't know exactly what went down, the factory reports that they have 3600 RJ45s sitting in their warehouse now.

FCC/CE pretesting

As we mentioned above, the first round of FCC/CE pretesting didn't happen when the factory said it would. Once we understood what was going on, the factory scrambled to buy a small number of RJ45 connectors to build test keyboards to submit for lab testing.

For reasons that still aren't entirely clear to us, the factory didn't send sample keyboards to the test lab until April 24.

On the 26th, the test lab told us that they were short some documentation: electronic schematics, an electronic block diagram, a copy of the Bill of Materials, a datasheet for the LEDs and an "Operational Description." We turned all of that around in a couple of hours, though we had to guess on the operational description and nobody had ever asked us for a block diagram for the keyboard before. The test lab promised to have results back in two business days, though this week's May Day holiday meant we didn't see the first round results until Tuesday.

We almost passed.

Of course, in the realm of certification, "almost passed" means "failed." The Model 01 had just a little bit too much electromagnetic radiation in the 35-115Mhz range.

Because this sort of thing is pretty common, and a little bit of a dark art unless you have a lot of experience, we designed the Model 01's circuit board to let us add slew rate limiting capacitors on the USB data lines.

One of the services the test lab offers is engineering support to help resolve exactly this sort of issue. The next step is for them to evaluate our proposed solution and make their recommendations for a remediation. It sounds like we'll have their feedback by the end of the week and then they'll test the solution next week.

If the next round of pretesting looks good, we can start building the electronics for the PVT units in a little over a week. We'll turn around and use some of those PVT keyboards for the official CE certification and FCC Declaration of Conformity.

Keyswitches

The factory currently has enough keyswitches in stock for the first 500 keyboards. (They've counted them twice.) Due to a mistake by their purchasing department, the rest of the switches didn't get ordered until February. Matias, our keyswitch manufacturer, initially thought they'd be delivering all of our switches sometime in March, but that didn't end up happening. They've committed to delivering us at least 1000 keyboards worth of keyswitches per month from here on in.

The delay is related to an improvement the they made to reduce the amount of "wobble" in the switches at the end of last year. The new switches feel a lot nicer, but due to the tighter tolerances, their manufacturing operation has taken some time to come up to speed. The team at Matias is bending over backwards to get us switches as quickly as possible and right now we believe that we'll have covered 90% of pre-orders with June's keyswitch delivery.

Keycap injection molding

When we last wrote, we told you that the keycap factory was reworking all of our keycap stems, so they didn’t grip the keyswitches so hard that they ripped the keyboard apart when you try to pull one out. That work was supposed to be completed by, roughly, the first week of April. It ended up stretching another two weeks. We got test keycaps in the mail at the beginning of last week and didn't get the factory's test report until the middle of last week. By our measurements, 48 of the keycaps required somewhere between 1.3-1.5 kg of force to remove, with the rest at 0.9 kg to 1.2 kg. 0.9 kg is way, way too loose. 1.2 kg would be workable, but isn't what we'd expected.

The factory's measurements all come in a bit lower than ours, with 20 of the keycaps at less than 1 kg of force being required to remove them and no keycap requiring more than 1.2 kg of force to remove. We think the difference can be chalked up to us using a fancy digital force gauge and them using something a bit more analog.

After a bit of negotiation, the factory agreed to have the keycap factory fix the keycaps that they measured as taking less than 1 kg of force to remove. They'd promised us an estimate of the time required to to modify the tooling last week, but we're still waiting on it. Thankfully this kind of modification is relatively quick and easy. To get the keycaps to fit more tightly, the plastic key stems need to be a hair bigger. And to do that, the factory needs to shave a little bit of metal off the injection molds. (To reduce the force required to remove the keycaps, they had to cut big holes in the injection molds, weld in new steel and remachine the molds. That was the bit that took so long last time.)

Keycap laser engraving

In mid-April, the keycap engraving supplier sent updated samples back to our factory. They'd done a pretty good job of getting the legends and paint in the right place, but even from the photos, we could see that there were a few labels that were still misaligned. The factory promised to have the laser engraving supplier take another pass at it. That new version of the keycaps was due to be returned to the factory last Wednesday, but never showed up. The factory is now promising that they'll be done sometime today.

We're starting to suspect that the only way these issues are going to get ironed out is to have Jesse go sit at the laser engraving supplier's office and work with their engineers. If that happens, you can expect a bunch of photos in the next backer update.

Center bars

In the last update, we told you that the factory said they'd resolved the issues with the center bars deforming a bit as they came out of the injection molds. They sent us samples. They had, in fact, resolved the curvature issue. From there, we pointed out a few issues we saw with the finish on the samples and asked them to spend a bit of time cleaning up the surface of the injection molds. At the beginning of last week, we got new samples in the mail. We're happy enough with the cleanup work they've done to declare the center bars "done"

Short RJ45 cables

One of the small, but important details we've been working through has been the RJ45 cables we use to connect the two halves of the keyboard.

The longer cables were pretty easy to sort out, but the shorter cables have been unexpectedly difficult. Most cable vendors didn't seem to get why we needed our short cables to be so short. (They have to fit inside the little cutouts between the two halves of the keyboard.)

We specced 12 or 13cm long cables, end to end. We got samples back that were half that length and a couple that were double that length. Some of the samples had connectors that were so long they couldn't fit inside the keyboard. One or two were so stiff that they put serious lateral pressure on the RJ45 jacks. That could lead to the jacks becoming less reliable over time. One sample had connectors so short that they got stuck inside the jacks. One sample had shrouds on the cable that were so long that the connectors wouldn't even fit into the jacks.

One vendor has made us a new set of sample cables that look like they're going to be just right. They match some samples that were made by a different vendor last year. Last year’s cables fit great and worked well, but weren't particularly well made. We expect to have the new vendor’s version in hand sometime next week.

USB Type C Cables

Last time around, we mentioned that we were looking at possibly forgoing the "right angle" Type C cables if we couldn't find a vendor who made something that seemed high enough quality. In the end, all the right-angle Type C cables we found that were well made had gigantic plastic housings around the connectors that prevented them from getting a solid connection to the Model 01. All of the good-looking right-angle cables were either mechanically or electrically wrong.

We've ended up deciding to go with a straight Type C cable from a vendor our factory has worked with before. It's still 1.5 meters long and our teardown shows it to be well made. It has a 56K pull-up resistor in the right place, which means it's not fraudulently claiming to support USB Power Delivery, so it shouldn't melt when you use it to charge your phone.

In the end, it's a pretty boring cable. We're ok with that.

Software

On the software front, there's lots of great stuff happening.

Algernon has been hard at work on Chrysalis, the cross-platform desktop interface for managing, programming and flashing the Model 01 (and other keyboards that run Kaleidoscope or have similar serial communications protocols.)

Over the past month, he's built functionality to autodetect a supported keyboard when you plug it in, flash new firmware on the keyboard over the USB line using @noopkat's avrgirl, the beginnings of a graphical LED theme editor, as well as a few more bits. He’s released a first “alpha” version which includes a mock keyboard, so you can try it out, even without a supported keyboard.

You can follow along with his progress on GitHub: https://github.com/algernon/Chrysalis, and can read a recent progress report on his blog.

Since Algernon has been focusing on the GUI, the firmware itself hasn't seen a ton of work over the past month. (It doesn't hurt that we have implementations of pretty much every keyboard feature we've reached for.)

The other exciting development on the firmware front isn't exactly Keyboardio related. About a week ago, Wez Furlong popped up with patches to Kaleidoscope that let him add support for keyboards with ARM microcontrollers and Bluetooth connections. One of his first targets is Adafruit's nRF52 Bluetooth board.

There's still a fair amount of refactoring to go before it's clean and easy to use Kaleidoscope with ARM chips and Bluetooth connections, but we're really excited to see this happening. If more keyboards can run Kaleidoscope, more intrepid hackers will be building cool plugins to make your keyboard better.

Scheduling

Our current best guess is that Jesse will be in Shenzhen in about two weeks to oversee the PVT run.

Sometime right around then, Kickstarter backers should expect to get email asking them to confirm their shipping addresses and keycap preferences. While we already have shipping addresses for post-Kickstarter preorder customers, we’re planning on sending them surveys, too. We know it’s been awhile since folks ordered and we want to make sure your Model 01 doesn’t end up shipped to the wrong place.

PVT assembly should take one week. After that, most of the PVT units will be express-shipped to customers. We don’t have a firm methodology for who gets the PVT units. Our plan is still to send those units to early Kickstarter backers located in the United States who confirm on their surveys that they’re happy to get some of the earliest production keyboards, and to put them through their paces, but we’re going to play it a little bit by ear.

A few will PVT units will be held back for reliability testing. The factory thinks that will take about a week and a half. If reliability testing of the PVT keyboards looks good, it's full steam ahead.

<3 j+k