A very very Happy New to everyone.

New year is knocking at our door steps and it is time to make gifts to our loved ones, and of course to yourself.

Apart from getting a new monitor, which I talked about before, I decided that it was not enough to satisfy my list of presents and decided to replace my HP hw6510 with HTC P3300 “Artemis”. I know it runs Windows, but it is the best as of today - plus I got there WiFi, Bluetooth, GPS and it is small and coool. [My New 2007 Year Tech Gifts]

This is the last post for year 2006 and I would like to wish to everyone a very Happy and Prosperous New 2007 Year and Happy Holidays.

Installation sources are wonderful things. That’s where the packages come from that we install on our SUSE machines.

So how do we add them in YAST? Is there a way to add them via the command line? Is there any way to automatically add them? Is there a list somewhere of repositories to add?

I would like to take on the task of answering each of these questions.

Adding installation repositories in YAST

This process is relatively painless, outlined as follows:

Open YAST.
Type in your root password.
In the window that appears, click INSTALLATION SOURCE on the right.
In the window that appears, click ADD.

HowTo

After four successful years of revolutionizing how content is shared in the real world, Lawrence Lessig, founding chairman of Creative Commons, announced his retirement as chairman of the board last week. Lessig passed the CC torch to Joi Ito, a venture capitalist from Japan. Creative Commons (CC) is a non-profit organization and is known for its Creative Commons copyright licenses. “We provide free licenses,” Lessig writes, “that mark creative work with the freedom the creator wants it to carry, so others can share the work, or remix the work, or both share and remix the work, as the author chooses.” On the popularity of CC, Lessig says that, “within a year of launch, there were more than 1,000,000 link-backs to our licenses (meaning at least a million places on the Web where people were linking to our licenses, and presumptively licensing content under those licenses).” In June 2006, this figure had grown to about 140,000,000 link-backs. “We have hit a stride, and more and more of the net marks itself with the freedoms that Creative Commons helps secure,” says Lessig.

Lessig is happy to be replaced by Ito, calling him the “perfect next chairman.” “The key to our success now,” he says, “is to fit this project within an overall economy of creativity. We want to support and protect the sharing economy; we also want to build tools that would help support crossovers into the commercial economy. That will take the sensibility and insight that Joi has demonstrated in his whole career.”

“The decision [for Ito to step up] was made at the last board meeting”, Mia Garlick, general counsel of Creative Commons, says, “so that CC can have the benefit of Joi’s leadership and to lessen Larry’s load whilst still having his vision, insight and involvement as CEO. Joi has already contributed considerably to Creative Commons with his experience and insights on the “sharing economy” and his tech and industry background.”

Ito recaps the growth of the CC licenses since his involvement. “When I joined the board in 2003, the licenses had been launched and the movement already had a great buzz of activity and goodwill around it. At the time, some products like Movable Type had already integrated Creative Commons licenses, but for the most part, CC was a movement of like-minded people with a vision. Since then, Creative Commons … has become a standard feature in major search engines, Web services, software tools and content libraries.” >>>>>

Ok, so you are a Linux user or a power user. The question then is what does it take to become a valid, omnipotent, root-enabled superuser? One potential answer is read the book How Linux Works, by Brian Ward and published by No Starch Press, by the last word of the last chapter you may or may not have been transformed, a wizard waiting to be born.

A superuser requires a consistent knowledge level across a wide array of topics. When I first started with Linux (I believe my first serious paid efforts were around ’94/’95), I learnt much by doing and interacting with the command line prompt. However, this method leaves you open to gaps in your knowledge. You may understand IPtables for firewalling, but do you necessarily have fine control over consistent packaging or accurate placement of binaries under the correct directory structure. Brain Ward’s well written commentary definitely covers all the necessary bases.

Book Review

For Edgy Eft UsersInstalling on Ubuntu 6.10,Kubuntu 6.10,Xubuntu 6.10 i386,amd64

Open terminal from Applications—>Accessories—>Terminal From terminal do the following enter each line and press enter

echo “deb http://www.getautomatix.com/apt edgy main” | sudo tee -a /etc/apt/sources.list

wget http://www.getautomatix.com/apt/key.gpg.asc

gpg –import key.gpg.asc

gpg –export –armor 521A9C7C | sudo apt-key add -

For Dapper Users

Installing on Ubuntu 6.06,Kubuntu 6.06,Xubuntu 6.06 i386,amd64

Open terminal from Applications—>Accessories—>Terminal From terminal do the following enter each line and press enter

echo “deb http://www.getautomatix.com/apt dapper main” | sudo tee -a /etc/apt/sources.list

wget http://www.getautomatix.com/apt/key.gpg.asc

gpg –import key.gpg.asc

gpg –export –armor 521A9C7C | sudo apt-key add -

Now you need to update your source list using the following command

sudo apt-get update

Install Automatix2 Using the following command

sudo apt-get install automatix2

HowTo on Automatix2

As a continuation of the previous post, here are 25 shortcomings found by Frank J. Ohlhorst when he reviewed the yet to be formally released Microsoft Vista OS. I have added my views which are enclosed in parentesis, alongside the Vista shortcomings.

• Vista introduces a new variant of the SMB protocol - (I wonder what is the future of Samba now…)
• Need significant hardware upgrades
• No anti-virus bundled with Vista
• Many third party applications still not supported
• Your machine better have a truck load of Memory - somewhere around 2 GB. (Linux works flawlessly with just 128 MB… even less).
• Too many Vista editions.
• Need product activation. (Now that is something you will never see in Linux).
• Vista OS will take over 10 GB of hard disk space. (With Linux you have a lot of flexibility with respect to the size of the distribution.).
• Backing up the desktop will take up a lot of space. (Not so in Linux)

More Reasons

BSD and Linux programmers have had a lot of success in creating drivers for new computer hardware in a timely manner, but much of their effort has been without the support of major hardware manufacturers. Intel, Marvell, Texas Instruments and Broadcom, though separate and competing entities, seem by one consent to prevent non-Microsoft operating systems from working properly with some of their most widely-used network chips. To find out more about this situation, I interviewed representatives from network chip manufacturers and programmers from free software operating systems. Their answers are below.

The problem

If you have not had any trouble with network drivers recently, you probably aren’t aware of the problem that this article addresses, so here’s a quick overview: The increasing need for higher network chip performance and lower cost of manufacturing has encouraged companies that create computer network chips to abstract software that was previously permanently stored on the chip. When this software is integrated with the hardware, you don’t have to think about it — it works without any extra effort, and all you need is a driver so that your operating system can interact with it. If hardware specifications are not provided, the device can be reverse-engineered to create a driver from scratch. Programmers generally do this by playing with the hardware registers until they figure out how to interact with it.

When the software is abstracted from the hardware, it changes from an invisible program stored on a memory chip into a file that must be loaded into the network chip’s memory through the driver. Generally this software (no matter where it resides) is called firmware. In this case the driver does not interface with the hardware directly; it only does so through the firmware. In this scenario it is impossible to reverse-engineer the hardware because it is essentially brain-dead without its firmware — all it knows how to do is load the firmware. Once the file is loaded into the network chip, then the hardware knows how to be a network device. Essentially the firmware is its own device-specific operating system. Programmers are unable to reverse-engineer this kind of hardware because the only registers they can play with are the ones that load the firmware; they can, however, reverse-engineer the firmware interface once it is loaded. That’s still not an easy thing to do, but even if it weren’t so complicated, many operating system developers don’t want to write their own device firmware — all they want to do is write a driver that can load and interface with it.

The first challenge for operating system developers is obtaining the right to distribute the firmware file, which some manufacturers will not allow without significant restriction. Firmware is not operating system-specific, so the same firmware file can be used with any OS, provided it has a driver that can interface with it. So the second challenge is creating such a driver, which requires firmware interface documentation. Not surprisingly, many of the same manufacturers that prohibit or restrict firmware distribution also won’t provide interface documentation.

This problem is most prevalent with wireless network chips, but it’s also creeping into the wired variety as well. Most notably, Broadcom has developed a new PCI Express 10/100/1000 LAN card that uses several discrete MIPS processors, all of which require proprietary firmware to be loaded. According to a network driver programmer I spoke with, they are employed as such: One of the processors is responsible for loading the firmware onto the other chips; one sends network packets; one receives packets; one tracks packet state to perform TCP offload assistance and various other things; one handles negotiating with the host CPU as it puts/removes packets in/out of the descriptor rings. In this case, all of the firmware fits into an 87k file, but firmware files can be considerably larger: Intel’s firmware files are just under half a megabyte, and the firmware for the Alteon Networks Tigon II network controller is in the vicinity of 2MB.

Unrestricted redistribution of firmware files is satisfactory for some open source operating system projects like OpenBSD, FreeBSD, and many varieties of GNU/Linux, but others like Fedora Core and Debian demand an entirely free software environment, so redistribution of the firmware without the ability to modify and distribute the source code is prohibited. The standard response to this from the Free Software Foundation is to reverse-engineer the device and provide free firmware. So even though it is very difficult — almost impossible — to do it in the absence of documentation, could such firmware eventually be reverse-engineered? The development team that works on the OpenBSD operating system has a lot of experience with reverse-engineering, but both project leader Theo de Raadt and OpenBSD network driver programmer Jonathan Gray agree that such work would be impractical. Of reverse-engineering firmware and the hardware that it runs on, de Raadt told me, “We can sometimes reverse-engineer how to talk to a device… some are worse than others… but imagine reverse engineering the firmware of 300-400 devices on the market today! Behind their little ARM/MIPS buses, they are a no man’s land of undocumented-ness and bugs; hundreds and hundreds of bugs created almost all by the realities of ‘time-to-market pressures.’” The issue, in other words, is made worse by hastily-designed hardware that doesn’t work as it should, and requires specific workarounds in the firmware and/or driver. Sometimes manufacturers provide patches or documentation for driver programmers; sometimes they don’t.

So instead of lobbying for documentation to write open source firmware, de Raadt would prefer to simply have the right to freely distribute necessary proprietary firmware files with his operating system, along with correct firmware interface documentation so that a driver can be created, and information from the manufacturer regarding bug workarounds. Many network chip manufacturers stubbornly refuse to grant these requests, however. Theo de Raadt told me in an email, “Our efforts to do more wireless involves a few approaches. We reverse-engineer what we can. We borrow from other people’s reverse-engineering lessons where we can, for instance, prism54.org is a Linux team, but their reverse-engineer work has resulted in knowledge which we can obviously use to write a BSD driver. And finally, we dialogue directly with vendors to get more free access to documentation, early access to hardware, or firmware distribution rights (sometimes there is just no other way). Some vendors (in particular Ralink or Realtek) will even give us hardware before it goes on sale. Some give us documentation, some give us code. But largely a lot of American vendors are still stupidly resisting giving anything. In any case, all these efforts together now mean that we have more wireless support in OpenBSD than all the Linux distributions. Maybe even combined!”

According to Jonathan Gray, the drivers that OpenBSD currently has that require firmware that vendors won’t let the OpenBSD Project distribute are:

• acx (4) - TI ACX100/ACX111 IEEE 802.11a/b/g wireless network device
• an (4) - Aironet Communications 4500/4800 IEEE 802.11FH/b
• atw (4) - ADMtek ADM8211 IEEE 802.11b
• awi (4) - BayStack 650 IEEE 802.11FH PCMCIA
• bcw (4) - Broadcom IEEE 802.11b/g (this driver is still under development and does not currently work)
• ipw (4) - Intel PRO/Wireless 2100 IEEE 802.11b wireless network device
• iwi (4) - Intel PRO/Wireless 2200BG/2225BG/2915ABG IEEE 802.11a/b/g wireless network device
• wpi (4) - Intel PRO/Wireless 3945ABG IEEE 802.11a/b/g
• malo (4) - Marvell Libertas IEEE 802.11b/g
• pgt (4) - Connexant/Intersil Prism GT Full-MAC IEEE 802.11a/b/g wireless network device
• uath (4) - Atheros USB IEEE 802.11a/b/g
• wpi (4) - Intel PRO/Wireless 3945ABG IEEE 802.11a/b/g wireless network device

And the wireless devices that either do not require firmware, or that have runtime firmware that OpenBSD is allowed to distribute:

• ath (4) - Atheros IEEE 802.11a/b/g
• atu (4) - Atmel AT76C50x USB IEEE 802.11b wireless network device
• ral, ural (4) - Ralink Technology IEEE 802.11a/b/g wireless network device (2nd gen 802.11 Ralink)
• rum (4) - Ralink Technology USB IEEE 802.11a/b/g wireless network device
• ray (4) - Raytheon Raylink/WebGear Aviator IEEE 802.11FH
• rtw (4) - Realtek RTL8180L IEEE 802.11b
• wi (4) - WaveLAN/IEEE, PRISM 2-3, and Spectrum24 IEEE 802.11b
• zyd (4) - Zydas ZD1211 USB IEEE 802.11b/g wireless network device

Intel requires that people who use its Centrino wireless firmware submit to a lengthy license agreement before downloading and using it. While the majority of end-users may just scroll down and click the “I Agree” link to get past it, the process is not quite so simple for free software operating systems, which would have to provide the same license hurdle for every one of their users, and agree not to modify any of the driver header code that Intel provides. Commercial desktop GNU/Linux distributions like SUSE, Mandriva, and Linspire already do this, or have independent distribution agreements with Intel.

When reverse-engineering works

Reyk Floeter and various other OpenBSD programmers managed to write a free replacement for the proprietary Atheros hardware abstraction layer (HAL) called ar5k or “OpenHAL.” The HAL isn’t firmware; it is a form of abstracted device driver that actually loads into the operating system kernel through a small amount of driver code. The big difference between a HAL and a firmware is where the code resides; if it’s on the device, it’s firmware, and if it is loaded into the operating system’s kernel, it’s HAL.

ar5k works with many Atheros-based wireless cards and has been examined and recently given a clean bill of copyright health from the Software Freedom Law Center. Mysteriously, it has virtually no support from the Madwifi Project, which is the development team responsible for creating Atheros drivers for Linux-based operating systems. Madwifi continues to primarily support the proprietary Atheros HAL, though there is an old and uninformative page in the Madwifi Wiki about it. The email addresses listed for the Madwifi developers either bounce or elicit no response to requests for comments on why there is no apparent effort to use the open source HAL in Madwifi. So if it exists, it’s free-as-in-rights, and it works, then why doesn’t Madwifi use ar5k instead of the proprietary HAL? It may very well be pressure from Atheros that keeps ar5k out of Madwifi, but no one at Madwifi or Atheros would talk to me about it.

The manufacturers speak — or don’t

In order to find out why network chip manufacturers are so polarized in their support of free software operating systems, I made contact with company representatives at Atheros, Intel, Marvell, Atmel, Ralink, Texas Instruments, Broadcom, and Realtek. Not surprisingly, the manufacturers who shun operating system programmers also seem to be reluctant to talk to the press.

Atheros

After weeks of repeated requests and pleas for any kind of response just to verify that the PR email address works, an Atheros representative told me that she was unable to find anyone at the company who was qualified and willing to comment for this article.

Intel

Intel punted me to different people a few times, then after a short delay and a bizarre inquisition into my professional background and “intentions” in writing this article, told me that the company had nothing to say on the matter of wireless firmware distribution rights and interface documentation. Considering Intel’s outstanding PR record and its general willingness to provide hardware documentation for the PCI chipsets and drive controllers that it makes, this behavior is unusual. One of the questions I asked Intel was if it felt its uncooperativeness with free software developers was in direct conflict with this presentation about the importance of participating in and supporting open source software development that Intel engineer James Ketrenos gave last summer. In it, Ketrenos says several things in favor of corporate cooperation with open source software developers, including these points:

• Enable the community to do as much as possible
• Only keep internal those things that the community can not contribute to (Example: Certification testing)
• If you need to keep IP closed source (for example some whiz-bang algorithm), document the hardware sufficiently that the community can provide their own.
• Treat the community as if they were a member of your internal team

Broadcom

The Broadcom press relations person I spoke with very much wanted to help me, but could not find anyone at Broadcom who felt qualified to comment.

Marvell

I have already written about the substandard responsiveness of Marvell’s outsourced PR agency (see the bottom of the linked article).

Texas Instruments

An internal Texas Instruments press relations person was very responsive to my requests for comment at first, then punted me to an outsourced PR agency which, in turn, ignored several emails asking for information. After more than two weeks of repeated inquiries, a representative of the PR agency informed me that she was unable to find anyone at Texas Instruments who was qualified to comment on TI’s policies on providing hardware documentation and firmware redistribution agreements.

Ralink Technologies

I had no trouble getting through to Ralink, where I spoke with company representative Lillian Chiu.

Ralink has been responsive to requests for hardware documentation without requiring an NDA. Why can Ralink do this when competing manufacturers such as Intel and Marvell require non-disclosure agreements?

Lillian Chiu: It’s our philosophy to spread the technology without border, along with high performance and low cost.

Because of Ralink’s cooperativeness with projects like OpenBSD and Linux kernel developers, Ralink’s products tend to be very well supported in so-called “alternative” operating systems. Do you see this as a competitive advantage? Does Ralink sell more network products as a result?

LC: Our customers have often provided positive feedback for our best-in-class device support. We see it as a win-win situation where advanced users get more flexibility while we sell more products.

If a programmer needs to access Ralink hardware documentation, whom at Ralink should they contact?

LC: Please visit Ralink Web site at www.ralinktech.com for details under /support/forums.

Atmel Inc.

Richard Bisset, product marketing director of the multimedia and communications group, offered several minutes of his time to answer some questions about wireless drivers and firmware.

Atmel has been responsive to requests for hardware documentation without requiring an NDA. Why can Atmel do this when competing manufacturers such as Intel and Marvell require non-disclosure agreements?

Richard Bisset: For some things we do require non-disclosure agreements, but we are generally able to provide the API documentation and the firmware driver interface specifications for our hardware. As to why others may not be able to do this… well, our software is developed in-house, but others might out outsource their driver development to third-party companies, so they may not even have the documentation that a programmer requests.

As to why Intel might be so secretive with this information if they do have it, I can’t say. Perhaps because more and more features are actually being moved from the firmware to the driver, some companies are being more guarded than others. Additionally, as the Centrino laptop processor includes wireless, it may be they don’t want to give up any potential IP that could help their competition — I don’t really know for sure.

Because of Atmel’s cooperativeness with projects like OpenBSD and Linux kernel developers, Atmel’s products tend to be very well supported in so-called “alternative” operating systems. Do you see this as a competitive advantage? Does Atmel sell more network products as a result?

RB: Well, if you look at the success of Intel, Marvell, Broadcom, etc. — the big players in the wireless industry — they are successful with the OEM and peripheral markets. As the embedded markets generally require more technical support, they probably don’t care much about getting another 50,000 or 100,000 units sold and can be tight with their APIs. We were late to market with our 802.11g products, so we missed the OEM opportunities, and with wireless, you’ve only got three real chances for success: you can be first to market with a technology, or you can have valuable and unique features that no one else has and the market wants, or you can have the lowest price. Atmel wasn’t first, didn’t have any new unique features, and wasn’t the cheapest, either. With the PC and OEM markets being somewhat locked out, we repositioned to focus on the embedded space where the market was experiencing and predicting large growth. In the embedded market, if you don’t get documentation to developers, then you both fail.What’s the big secret with device firmware, then?

RB: Firmware is generally running on the chip itself, as opposed to the host. The code is written tightly coupled to the chip it is running on and implements/enables functions that the hardware supports. Therefore, there generally is a lot of IP within the firmware. Our firmware is loaded from an external flash into internal SRAM, but more and more semiconductor manufacturers are moving firmware functions to the driver.

We usually provide driver source code, and we try to put it under the GPL if possible, so that’s usually good enough if you want to write your own driver. If you want to see more than that, we generally require an NDA, or if you’re an embedded customer, we provide reference platforms.

So the firmware is so secret that you have to sign an NDA to see it, but not secret enough that some stranger who works on an obscure operating system can see it with merely a signature?

RB: It is kind of a strange situation, yes, and sometimes we run into people who are reluctant to sign NDAs. It really depends on the customer and target application. We don’t require an NDA to see hardware APIs, and I think that’s what you’re talking about with regard to documentation.

If a programmer needs to access Atmel hardware documentation, whom at Atmel should they contact?

RB: We have contact forms through our Web site where you can make requests, and they are directed to the proper person at Atmel.

Realtek Inc.

After a few emails, I made contact with Tracy Ho, a Taiwan-based representative for Realtek.

Realtek has reportedly been responsive to requests for hardware documentation without requiring a non-disclosure agreement (NDA). Why can Realtek do this when competing manufacturers such as Intel and Marvell require NDAs?

Tracy Ho: For some years Realtek has been one of the largest NIC/LOM solution providers. One of the major reasons for this is that Realtek takes customer service as one of its top priorities. We aim to work with our customers as partners in a mutually cooperative environment. Our product sales and technical support teams are well-recognized by our customers and we strive to provide them with the convenience of flexibility and real-time support. One of the ways we do this is by releasing general hardware documents without requiring a NDA.

Because of Realtek’s cooperativeness with projects like OpenBSD and Linux kernel developers, Realtek’s products tend to be very well supported in so-called “alternative” operating systems. Do you see this as a competitive advantage? Does Realtek sell more network products as a result?

TH: Realtek has been working very closely with various operating system providers pretty much since the company was first started, and we do consider this to be one of our competitive advantages. Over the last decade, the great expansion of networked devices throughout the home and work environment, and the use of open-source operating systems in many such devices, has definitely helped us expand market share.

If a programmer needs to access Realtek hardware documentation, whom at Realtek should they contact?

TH: For most hardware documentation, programmers can access Realtek’s website. Any other questions can be directed to our technical support teams (contact information available from the “Contact Us” link on our website).

All we need to do is make sure we keep talking

Some of the non-responsiveness of manufacturers may just be bad PR work, but the same companies that wouldn’t talk to me have also refused to reply to free software programmers who have requested the same information. The impression I got from most of these companies (excepting Intel) was that they were not at all prepared to deal with the issues of firmware redistribution rights and hardware API documentation requests. That they have ignored free software programmers’ requests is not necessarily a sign of unwillingness to participate, but perhaps a general sense of confusion as to how they are able to help. No one seems to know whom to talk to at the company, and in some cases the proper documentation may not exist — or it may belong to yet another company that the hardware manufacturer outsourced the firmware development to.

On the other hand, it was just as difficult to contact and get comments from the Madwifi developers, and it seems that each individual wireless driver in the Linux kernel has different people working on it. Even if one of the above companies wanted to provide the appropriate materials to create free drivers and firmware, how would they know whom to contact? The irony in this story is that the heart of the problem lies in a lack of communication, but not between operating systems and network devices — between hardware manufacturers and the software developers who are trying to support their devices.

Jem Matzan

Shifting from 19.4% for the week of the 8th to the 14th of April 2006, Firefox user share rose to 23.2% in the week from November 20th to November 26th 2006:

 Complete Survey