Darwin Streaming Server 6.0.3 and Solaris

Apple has released a new version of their streaming software: Darwin Streaming Server, DSS.
Many production systems use Solaris 10 for DSS without a problem. DSS works nicely on Solaris 8,9 and 10 and some examples are already presented:

- DSS 5.5.x on Solaris 10 with ZFS and zones
- Prune's patch

Now we have a new version out: 6.0.3 with new features and better Solaris 10 support, here
For all of you interested in streaming you might want to check it out.

Wish list:
- OpenSolaris integration: IPS package, testing
- DTrace probes for DSS
- DSS 64bit support for Solaris
- Better documentation for OpenSolaris, Solaris

Enjoy and happy streaming on Solaris

Turbocharging your sysadmin day

It is final: Neil will visit Finland, to teach us about PDQ, Capacity Planning and Performance Analysis.
You don't get these sort of classes from your local vendor, so all admins pay attention !

Why Capacity Planning matters and why should you listen and understand such things ?

• don't end up having your Java development team changing your DataCenter infrastructure every 6-12 months based on their fantasies how the world should run and support their latest builds. More and more visible is the penetration of a dozens of Java architects, developers and consultants into the heart of Data Centers. They change wires, servers, standard operation procedures to make the applications faster and scale, so they said. Instead they focus in making their own J2EE application less buggy they start shifting and virtually touching intimate parts of the data centers without a clue what are they doing. Management pressured by business demands accepts quickly and approves all this mess without a real understanding. At the end: admins have to clean up after all these folks and redo all the things to avoid severe service outages (Java is an example here based on the high number of current projects, same applies to any other language or technologies)
  
• avoid the mess - where everyone is a system manager and nobody does know what's going on
• save time and money
• work smart
  

One might argue that hardware is anyway so cheap nowadays so you can entirely skip this section and renew all over again your infrastructure when you run in troubles... and here comes the Guerrilla Capacity Planning class:

• Capacity Planning and mathematics. Capacity planning is not a software or something you buy from amazon.com or your local software store
• Tactical planning
  
• Analyse your current load capacity
• Model your business and forecast
• Avoid spending $$$ for another commercial software, PDQ the open-source analytic solver software can be easily used to model and resolve performance problems

Welcome, Neil

JVM Internals

Good morning !

I have posted a new training material about Java Virtual Machine Internals + a real example of using DTrace to debug a sample application delivered with BEA Weblogic Portal 10. You can find the material and source code under this address.

Why JVM Internals ?

As SysAdmin you have to handle and maintain dozens of services running on top of an
application server: BEA, Sun, IBM, Oracle, etc ... but most often you
find yourself in troubles understanding concepts like: garbage collection,
generations, heap size, object allocation. The presentation will help you to understand
why is important to size a JVM and how to do that, to troubleshoot
a JVM problem on Solaris or simple debug a Java application using DTrace.
A real case example has been used, more will come later on.

If you're interested to have a class about this topic let me know. (Scandinavia only)

Go Norway !

Just returned from my summer holiday, from Norway. Some photos available here:
http://www.nbl.fi/~nbl97/trips/2007_Autumn_Nor/album/index.html

The trip was very nice. In aprox 3 weeks I made around 7500km with my Subaru. The weather was in general ok, sometimes raining, it was even snowing up in mountains, but in general cool :)

I have updated my photo gallery using JAlbum, very easy to use.

Enjoy!

Java for SysAdmins

Im a SysAdmin, taking care of all our UNIX installations, maintenance, software installation, security, performance problems and all the other bits, making sure all our services are up and running nicely.

This year was a very busy one with lot of changes in our datacenter: we moved to Solaris 10, started to use zones and learn DTrace. On top of these things we needed to ensure all our services were up and running no matter what. And here comes the fun. Most of our services are build as J2EE applications running on top of a application server. Lately we started to see a lot more alerts than we used to have... And it is not really fun when it is 3AM and your site does not work. I had the painful job to debug and try to find out whats wrong with these services. Administering an application server is not exactly the same you would look a sendmail server or a busy database server. Things are a bit different and you will need proper lenses to observe the things.

Be prepared cause more and more Java applications will affect your SysAdmin life :)

Major problems:

• A lot of abstractions. You have the OS, the JVM, the app server, the application and dozen of other 3rd parties components used by the application. To debug all this stack is a pain unless you are setting the proper glasses to see where the problem comes from. Prepare yourself with a lot of patience.
  
• JVM Tuning: Most of our J2EE applications were simple deployed to PROD env without a proper planning and tuning phase. I have discovered that the JVM layer was simple ignored - running on default options. As well it was somehow confusing who should be in charge of tuning the JVM: the Java development team, or our support unit. We ended up using a cooperation of these two: development and support teams.
• JVM core dumps: sometimes you will see what you will never expect: the entire application goes holiday, your JVM simple dies with a core dump. What you gonna do next !?
• Outsourcing: it is far more complicated in 2007 to find out who wrote that component or module, who has integrated into the core product or who has correctly ran a regression against the new version of the product. As teams are spread around continents it is far more important the documentation part, which for majority of the projects is almost non existent.
  

Fixing the problems. A long pain process in order to get all these things fixed and a lot of hours spend on issues not very well orientated as system administration. Some issues are more process orientated, some other are simple technical standard procedures which needed to be in place and proper documented. Here are some of them:

• JDK versions: it is not evil to keep up to date your JDK versions. Some folks have the feeling that they shouldn't change at all the JDK version delivered by the app server vendor. So lets say you use Java 5, like jdk 1.5.0_06 so make sure you are running the latest _12, for instance. Setup a process to keep up to date the JDK. Same applies to jdk 1.4.2, or jdk 6.
  
• JVM tuning: I started to setup a simple procedure to be able to sleep and have a peaceful night. Check it out
  
• Debug tools - We needed smart tools in operations: pstack, jstack, prstat, dtrace, plockstat will save your day a bit if you know how to use them against a JVM. Remember, your application looks like a big and fat 32bit or 64bit process. You gotta look inside and see where the problem comes from. Checking as well the app server log is another way to solve the problem, but very much depends how much trash the log contains and how abstract it is.
• Java 5 or 6 ? It seems a bit complicated to simple just move from Java 5 to 6. Your app server vendor might not like this so check it out before trying it. We are stuck in Java 5 and Java 1.4.2.
  
• Thread dumps - A normal operation procedure: generate a thread dump, run a 'kill -3 appserver_jvm_pid' and examine closely the thread dump. when your application looks like dead most likely this is the right time for it.
• prstat -mL -p appserver_jvm_pid - will set you free. You can easily see each lwp and the CPU consumption, LAT, LCK. A good way to see what your lwps are doing. prstat is more capable comparing with top.
• plockstat - a new guy in town. Based on DTrace this will report user level lock statistics. Very useful in case your application seems to do nothing. We are currently developing some scripts using it.

As many of you out there are starting to administer more and more J2EE applications you gotta setup a plan and work smart in order to be able to sleep. Try the JVM tuning procedure inside your project and let me know how it went. I bet many of you are already using some sort of JVM tuning procedure, I would be glad to hear opinions and comments.

Peaceful summer,
Stefan

Subaru AWD system...

For Subaru fans and not only: http://www.youtube.com/watch?v=Q7dVFY5CxT0

Darwin Streaming Server on Solaris

I have been very busy with my job so no entries for a long time. I have helped some friends of mine with Solaris and Darwin Streaming Server. I saved all instructions together in a short doc. This is work in progress ! Enjoy.

http://www.nbl.fi/~nbl97/solaris/media/streaming/index.html

Happy Day for all SysAdmins

Happy SysAdmin Day !

Take a short break, get some cake and some drinks. I opened a nice 12 years old scotch whisky MacAllan bottle and some cake... even there was not enough space for 'SysAdmin' string the surprise was nice. Thanks Manta !

For all SysAdmins folks have a good day and for all of you which dont know what this day is, please have a look:

http://www.sysadminday.com/
The Song

Educational DTraceToolkit...

I've been busy with DTrace and Solaris. In fact I was working in getting ready a DTrace presentation for
Opensolaris/DTrace community: a generic workshop about DTrace and DTraceToolkit - a simple introduction on the new dynamic tracing framework and as well a presentation of DTraceToolkit with real case examples.

You can call this the first version, many new things planned to be added here:
- DTrace/DTT for Database Administrators
- DTrace/DTT for Java Developers
- Other examples using DTT
- support for certain local groups: romanian, etc

The main point of this presentation was to bring to people's attention the toolkit: a big collection of D/sh/ksh scripts used to help you in troubleshooting and debug situations. Please feel free and reuse the presentation as you like. Im talking with Brendan to see if we can include it under the toolkit or find a better place for it.

At the moment you can download the presentation from this address

I think it is very important that people understand what new tools, mechanisms Solaris 10 is providing. Working for Sun Microsystems Finland I must ensure that our local customer base does get the real signal about DTrace and how can they use DTrace in production systems. So here comes the point of presenting DTrace and DTraceToolkit.

Even if you are a bank, a telco, a software house or maybe a game company sooner or later you will have to observer closely your business and troubleshoot it when some things might not function correctly. Or better you have to look the entire system and see the entire stack how it functions... Years ago, call this the stone age, people used to simple restart applications or simple reboot the entire system when something was not properly functioning...In present we can still see such things happening. You have a choice using Solaris/DTrace to stop doing this and start understanding what's going on...

If you are in Finland and need to meet and talk about these things don't hesitate to contact me! Otherwise drop me an email, I really much appreciate your feedback.

Cheers,
Stefan

Testing DTraceToolkit

I have started to help Brendan to test every DTraceToolkit script and write some more documentation about DTT. Why ? Well simple put it because DTraceToolkit (call it DTT) is already a big toolkit with lots of interesting and smart scripts ready to be used within companies and businesses I would say. What I was looking for was to make sure the DTT was tested and 'certified' for a certain Solaris version release.

So, looks like Im in charge of designing some simple test procedures for DTT scripts. First of all I should be able to create an automated script which will test every DTT category and report several numbers back: time spent to execute the script, if the script was executed fine or not. Doing this we could ensure that DTT works fine on a newer version of Solaris Express or S10. In the same time I started to record all defects or RFEs related with these and document and submit to Brendan. We might need to think for a better bugtracking system...

So this way I started to write dttck - The DTraceToolkit Checker. For fun this is what dttck does at the moment:

sparvu@earth>./dttck
Check DTrace privileges - ok
Usage: dttck [options] [script...]
options:
    -A All scripts
    -a apps scripts
    -c cpu scripts
    -d disk scripts
    -e extra scripts
    -l locks scripts
    -m mem scripts
    -n net scripts
    -s system scripts
    -u user scripts
    -r regression
    -o output report
    -q quiet mode

The script can be used to select certain categories or run individual scripts. Example below how Im testing the Apps category:

sparvu@earth>./dttck -a
Check DTrace privileges - ok

Category: Apps
Scripts: all
Date: 2006-01-26 22:20:04
Number of scripts: 4
     script: httpdstat.d - ok
     script: weblatency.d - ok
     script: shellsnoop - ok
     script: nfswizard.d - ok
Elapsed time: 16 sec(s)

The entire Apps category was tested by firing each script. The checker does record the trace for every run under a report directory called something like: hostname_datetime. E.g.: earth_200601262220, where every script will record its execution trace. At the end a final report file would be compiled gathering all data:

sparvu@earth>ls
Apps.httpdstat.d.out Apps.shellsnoop.out report_earth_200601262220
Apps.nfswizard.d.out Apps.weblatency.d.out

sparvu@earth>cat report_earth_200601262220

REPORT QA DTraceToolkit

DTraceToolkit version 0.92, 12-Jan-2006
Operating System: SunOS 5.11 snv_31 i386

Category: Apps
Scripts: all
Date: 2006-01-26 22:20:04
Number of scripts: 4
     script: httpdstat.d - ok
     script: weblatency.d - ok
     script: shellsnoop - ok
     script: nfswizard.d - ok
Elapsed time: 16 sec(s)


Summary Report

Categories: 1
Failed Scripts: 0/4
Passed Scripts: 4/4
Total Elapsed time: 16 secs

Report generated by: sparvu

There are many things to fix and refine in DTT. I think that DTT is a powerful toolkit which should be used within companies. That's why Im dedicated my time testing and documenting various parts of it. There are no such toolkits under Windows, HPUX, or any other Linux based systems making Solaris a very nice platform for production and development.

From Santa Claus Office: Merry Christmas !

From Finland, Santa's home, a nice Xmas everyone !

We are heading fast to 2006 but we should not forget that 2005 was a good year. Lots of goodies were send out:

• Solaris 10 and the new Galaxy servers
• Studio 11
• ZFS
• A new era: CMT and the new UltraSPARC T1 processor
  
• Announcements: Oracle, PostgreSQL
• The BrandZ Community
• Many contributions to OpenSolaris and many other new things.

The "Unlocking the Secrets to the Scalability Puzzle" seminar was a nice dialogue with several Sun engineers about how well Solaris scale and how can you make your application scale better. Check it out it has a lot of good information. Thanks to Richard for explaining and adding some good points why Solaris matter.

On the Java front: for all of you which are still running your businesses under 1.4.2_xx VM a good news is that jmap was backported to the 1.4.2_09 ! So you don't have to change your application server from 1.4.2 to Java 5 to use jmap. Try as well to read Sundararajan's blog about jmap, FatCatAir's entry about Safepoints and as well Alan's blog about the great Java error: OutOfMemoryError !

And if you got tired or you want something new and nice book a room to SnowCastle and start your compilers, you won't need extra coolers :)

See you guys in 2006 !

Growing the OpenSolaris community

This week, I think, it was a really good week: lots of 'goodies' were published: ZFS, Studio 11, PostgresSQL and Oracle press releases, Nevada build 27. Got as well my new books from amazon.co.uk:
TeX, XML and Digital Typography and Digital Typography using LaTeX. And on top of these things had my birthday anniversary: Im 31 years old now...

Try the new ZFS and have a look on the OpenSolaris community. Experiment with Express and see if you like it. Post your findings ! Sun made a significant effort to build a Solaris community and I think this is really working well.

Being tired of Linux: try Solaris Express !

Nice weekend everyone !

ZFS demo, check it out

A very nice demo made by Dan Price: http://www.opensolaris.org/os/community/zfs/demos/basics/
Check it out !

The ZFS Team sent some word about the ZFS on opensolaris forum. Check it out from this address: http://www.opensolaris.org/jive/thread.jspa?threadID=3594&tstart=0

Nevada b27 does include ZFS. You better hurry up and start your download !

Cool guys !

Studio 11 free...

Good news for developers: Studio 11 tools are free ! check it out ...

This is a huge step forward after the release in summer 2005 of Studio 10 free for OpenSolaris project. Good work Sun !

References:

http://www.sun.com/smi/Press/sunflash/2005-10/sunflash.20051026.1.html
http://www.sun.com/software/products/studio/index.xml

Sun's Galaxy servers

I have to say the new Galaxy servers from Sun look cool. I wish I could have one or two around to do some tests. I hope more and more folks will turn to use Galaxy AMD Opteron servers instead using Dell.

I am running Solaris... and I like that

Notice: No flames please ! I'm not interested in starting any debate about your favorite OS or why you like Debian distribution or any other Linux based distro.

I would like to share my ideas about Linux based distributions and why I keep using Solaris. More and more folks are talking about Linux showing its merits. There is no such operating system like Linux, but instead is just a kernel developed by a group of people which are publishing their work under: www.kernel.org. Some of these folks are working for different organizations and companies: OSDL, IBM, RedHat, Novell etc. A kernel cannot be very useful without the entire collection of tools and utilities around it: the distribution, or simple the distro. Such distros are: Fedora, SLES, Gentoo, Debian etc. Behind these distributions are individuals or even commercial companies hiring people doing the work. Example being here: RedHat, Novell. Some of these distros have guys working in QA by testing their bits in some other cases nobody does that. So a very diverse and big community around Linux.

I do believe Linux based systems can easily coexist with Solaris and other systems: OpenBSD, Windows, NetBSD, FreeBSD etc. However I hear more and more talking and saying: just replace this with Linux. It would just make your life so easy and solve all your problems... etc. Hm, why everyone should convert from OpenBSD or Solaris to Linux based systems !? Sometimes some IT magazines are presenting similar message out there: convert or die ! Why not sharing and coexist ? Torvalds believes Solaris users should die: "A lot of people still like Solaris, but I'm in active competition with them, and so I hope they die,"Well I hope they will do just fine. I never understood his aggressive politic against Solaris... Is Torvalds afraid ?

However I have to say my point of view on this matter, about the Linux based systems. I had some experience with such systems since 1997, in Romania, when I was finishing my degree at www.ugal.ro. I had access to Linux at that time and tried the system at home. It was kind of an experiment and it was running on my home PC. Soon I have invested my money in some CDs from Pacific HiTech Inc. distributing applications, the kernel, X11, kernel patches etc. Kind of fun! At the same time at school I had access to Solaris 2.5. After I left Romania, in 1998, to Finland my contact with Linux was kept by using SuSE and RedHat sometimes. At some point in time around 2000 I have totally gave up on Linux based systems. I have tried several times for curiosity Debian and Fedora too.

At the moment I do run OpenBSD, Solaris, and Windows XP. I do like Solaris and starting with Solaris 10 an entire new world has just opened: www.opensolaris.org ! Check it out ! Thanks to all Sun managers making this possible !

I do like Solaris more than Linux based systems and I consider Solaris much advanced and mature than any of distros. Some points before converting to Linux:

• What distro ? Fedora, SLES, RedHat Entreprise Linux WS, or AS or maybe ES because Oracle recommends that or maybe Gentoo to stay cool or maybe Debian to get lot of packages or Ubuntu etc etc Blah!. A lot of options. Nice. But why RedHat had to make 3 or 4 versions of their based Linux distro don't know. Same thing for SUSE, currently Novell. The same thing what Microsoft does with Windows... I would like to have the choice of having one distro called RedHat Linux. As simple as it is. Is there any Solaris AS, ES, WS , A1, B2, Solaris Entreprise ... nope it is just Solaris ! Dude get a life !

• I like what Eric Schrock mentioned in his blog about Solaris vs. Linux: Reliability, Serviceability, Observability and Resource management. Try to read the blog and think clear, don't fall on any of the sides

• A stable ABI does it matter !? I do believe that having a stable ABI would help vendors like nVIDIA and others develop things which will stay stable. Someone will say but hey, everything should be open sourced and it is just a matter of a recompile ... Everything ? Still waiting to see Oracle database server source code... Well I bet there will be always something which will not be open sourced so get along with it. So coming back having a stable ABI: Linux kernel does not follow this principle instead it does suggest that this is bad. Interesting point. I would be glad to see from RedHat, IBM or any other distro vendor a white paper on this mater: why having a stable ABI is bad and what are they recommending about that !?

I try to stop now, the list could go on but I do think there is space for all OSes out there so nobody must die :)

SAS: Serial Attached SCSI presentation

I've just read Rich's blog and found out some interesting presentation about SAS:
http://specials.seagate.com/sac/sun/

I'm thinking what vendors have already SAS controllers ? LSI, Adaptec ... !?

A new workstation runs... solaris express b25

Got a new machine: AMD X2 Dual Core 4800+, 250GB SATA2 disk, 2GB RAM
for aprox 2000e from MikroMafia. It is fast. Very fast. Worked fine with
Solaris Express build 25 from www.opensolaris.org/os/downloads

Some details below:

Mobo: ASUS A8N-SLI Premium S939 NF4 SLI 2*GLAN
CPU: AMD ATHLON 64 X2 4800+ S939 (2 x 1024 KB L2)
Memory: 2048 MB 4KPL 512Mb Kingston PC-3200 DDRAM
HDD: 250 GB SATA2 16MB Cache
Video: GEFORCE 6600 256MB VGA+DVI+TV-OUT

Disk:
SATA2 250GB Maxtor 16MB cache: works fine using nvidia controller.
This mobo has two SATA controllers. I haven't tried using Sillicon Image,
the 2nd controller.

genunix: [ID 640982 kern.info] IDE device at targ 0, lun 0 lastlun 0x0
genunix: [ID 846691 kern.info] model Maxtor 6L250S0
genunix: [ID 479077 kern.info] ATA/ATAPI-7 supported, majver 0xfe minver 0x1e
pci: [ID 370704 kern.info] PCI-device: ide@0, ata2
genunix: [ID 936769 kern.info] ata2 is /pci@0,0/pci-ide@8/ide@0
genunix: [ID 773945 kern.info] UltraDMA mode 6 selected
gda: [ID 243001 kern.info] Disk0:
ata: [ID 496167 kern.info] cmdk0 at ata2 target 0 lun 0
genunix: [ID 936769 kern.info] cmdk0 is /pci@0,0/pci-ide@8/ide@0/cmdk@0,0
unix: [ID 190185 kern.info] SMBIOS v2.3 loaded (2226 bytes)

Video:
I have a Geforce 6600, works fine with Xorg. I will try soon to install nvidia drivers
to gain some performance. Maybe I should try bzflag then :)

Network:
I have two Gig NICs. Im using the nvidia one fine without any
problems. However the second one, which is a Marvell 88E81001 should
have drivers for Solaris. I haven't got time for that yet.

lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
nge0: flags=1000843 mtu 1500 index 2
inet 192.168.1.5 netmask ffffff00 broadcast 192.168.1.255

gld: [ID 944156 kern.info] nge0: nge V1.1: type "ether" mac address 00:13:d4:53:db:f9
pci: [ID 370704 kern.info] PCI-device: pci1043,8141@a, nge0
genunix: [ID 936769 kern.info] nge0 is /pci@0,0/pci1043,8141@a
nge: [ID 801725 kern.info] NOTICE: nge0: link 100Mbps Full-Duplex up
pcplusmp: [ID 637496 kern.info] pcplusmp: pci10de,57 (nge) instance 0 vector 0x15 ioapic 0x2 intin 0x15 is bound to cpu 1

AMD X2 CPU:

0 on-line since 11/09/2005 18:51:15
1 on-line since 11/09/2005 18:51:17

Status of virtual processor 0 as of: 11/09/2005 19:49:54
on-line since 11/09/2005 18:51:15.
The i386 processor operates at 2412 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 1 as of: 11/09/2005 19:49:54
on-line since 11/09/2005 18:51:17.
The i386 processor operates at 2412 MHz,
and has an i387 compatible floating point processor.


The system is fast, very fast. JDS seems flying. Looking forward to get build27 soon.

BigLib...

BigLib should be a collection of many useful articles, tutorials about Solaris. At the moment I have published some of these under: www.nbl.fi/stefan.parvu
I will try to publish from time to time some some new things about: DTrace, ZFS, Solaris 10, OpenSolaris. I hope just to find the time to do that.

Welcome to my blog !

Hello world. My name is Stefan Parvu, Im a sysadmin working for Sun Microsystems Finland.


Notice:

All comments, infos I make are my opinions. They are in no
way related to my employer, Sun Microsystems.