Tuesday, September 9, 2008

InterBase 2009 comes with multicore support

I have just read the following news article on SDTimes:
InterBase 2009 comes with multicore support
A couple of aspects to this story seeme a little peculiar, while some comments require some clarification.

"Thus the update for InterBase 2009 is relatively subdued"

correct, I do not see any major new features in this new version, however a couple of new features highlighted are

"such as over-the-wire encryption and AES as an encryption scheme".

If you want over the wire encryption for Firebird you could use Zebedee. There have been a number of papers re. this in the past. However this is something that could be implemented in Firebird fairly easily if we so wished.

In terms of database encryption there is code to allow it now, however the original implementation was not reliable. We did put together a more detailed proposal, but after some lengthy discussions it was decided that such functionality was going to be too complicated for the majority of users, and most of them didn't want it any way.

I do have trouble working out what

"The biggest change, perhaps, is the removal of any non-symmetric multiprocessing (SMP) compatible versions of the database. All versions of InterBase will now be able to use multiple processors and cores."

means.. CodeGear have been stating that InterBase was SMP enabled since InterBase 7.0. So what does this mean? Was InterBase non SMP capable/SMP friendly on some platforms? Was there code that wasn't SMP enabled? Has the code been cleaned up, so that all the old platforms have been removed. Or is it just a licensing issue where the new SMP license now supports up to 8 cores/CPUs, while previously you had to buy extra CPU licenses?

One particular quote was very confusing

"For the existing open-source Firebird database, which is based on InterBase, these changes won't trickle down; that free database does not include SMP, disaster recovery or high availability features."

Firebird was originally based on the open sourced version of InterBase 6.0, but the code bases are now quite different after 8 years of independent development work. So to state that Firebird is based on InterBase, although this is technically correct, it is a little disingenuous.

Stating that changes in InterBase will not trickle down to Firebird is correct. InterBase as per Inprise/Borland/CodeGear/Embarcadero is a closed source database. After the code was originally open sourced on SourceForge by Borland, it was forked and Firebird was created. InterBase was then "closed sourced" by Borland, and development work has continued on InterBase in a closed source manner. No code or feature at any time has come from InterBase to Firebird. Although it is possible that Firebird code may have made it into InterBase. However we have no way of verifying that.

In terms of SMP, by re-introducing the process based Classic Arhitecture, Firebird does support SMP. However a process based architecture isn't multi-threaded. That's an issue that's being addressed in Firebird V2.5 (SuperClassic) and Firebird 3.0.

I assume by "Disaster Recovery" this refers to "Journaling with Write Ahead Logging (WAL)" something that has not been implemented in Firebird. I am unsure what the other "High Availability" features are.

Finally

"In October, he said that InterBase ToGo will be made available. Normally, InterBase has a memory footprint of around 100MB. ToGo will be slimmed to around 3MB"

I am lost on this one. I am sure by this he means a disk footprint, not memory. And most of the size of the normal disk footprint is taken up by a stupid installer (imho) and docs. My assumption is that InterBase ToGo is effectively a variant on Firebird Embedded, but since ToGo will not be available until October it is difficult to comment.

Thursday, September 4, 2008

Solaris 10, Firebird SuperServer and the Service Management Facility

I recently released Firebird V2.04 SuperServer for Solaris 10. This new build uses the Solaris 10 Service Management Facility to stop/start Firebird. Here are some notes on how it works.
I created the following manifest file (it can be found in /opt/firebird/ as gds_db-tcp.xml):

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
Service manifest for firebird
-->

<service_bundle type='manifest' name='firebird'>
<service
name='application/firebird'
type='service'
version='1'>
<create_default_instance enabled='true' />

<exec_method
type='method'
name='start'
exec='/opt/firebird/bin/fbmgr.bin -start'
timeout_seconds='10'>
<method_context>
<method_credential user='firebird' group='firebird' />
</method_context>
</exec_method>

<exec_method
type='method'
name='stop'
exec='/opt/firebird/bin/fbmgr.bin -stop'
timeout_seconds='10'>
<method_context>
<method_credential user='firebird' group='firebird' />
</method_context>
</exec_method>

<template>
<common_name>
<loctext xml:lang='C'>
Firebird
</loctext>
</common_name>
</template>
</service>
</service_bundle>

This manifest file is automatically installed by the pkg installer.
You can manually stop and start firebird using the following commands, instead of using fbmgr.bin

svcadm enable svc:/application/firebird:default
svcadm disable svc:/application/firebird:default

You can verify whether the Firebird "service" is running by using the following:

svcs -a | grep firebird

Supposedly the new Service Management Facility provides automatic recovery from software and hardware errors, so theoretically SuperServer on Solaris 10 no longer has need of the Firebird Guardian, since SMF should do the same job that the Guardian was designed to do. This is going to need some testing ....


Monday, July 21, 2008

Solaris 10 and iscsi

Seems that applying all of the patches discussed previously has broken my version of Solaris, as it now seems to have problems booting.

It seems to hang during the boot cycle, so I had to do some hunting to try and find out the cause.
Boooting with a -v (verbose) shows that the system is hanging in iscsi.
After some googling it seems the only way to fix this, so you can get in is to do the following..

1. Boot using -m milestone=none
2. Login as root
3. svcadm enable -r filesystem/minimal
4. svcadm enable -r filesystem/local
5. cd /etc/iscsi
6. mv iscsi_v1.dbc iscsi_v1.dbc.bad
7. reboot

Now you can get in, I suggest you go and install the latest iscsi path from sunsolve:
Patch number 119091 for x86 Solaris.

reboot and hopefully all will be well.

Wednesday, July 2, 2008

Solaris and the 256 File Descriptor Problem

Some of you reading this may remember that back when InterBase 6.0 was released on Solaris we ran into a fundamental problem on Solaris. It seems that since 1992 there has been an open bug documenting Solaris' file descriptor problem: "32-bit stdio routines should support file descriptors >255."

Bug id:
1085341

When this issue was discovered, we chose to use SFIO, a stdio variant from AT&T
Comments on this issue can be found in jrd/common.h which forces a superserver build to use our own internal stdio.h found in extern/SfIO/include.

Well it looks like in Solaris 10 Sun have finally fixed the issue.
More details can be found in the article entitled:
Solaris OS Solutions to 32-Bit stdio's 256 File-Descriptors Limitation by Giri Mandalika dated May 2007

Suffice it say - the bug has been fixed, but in a fairly typical Sun way.... the fix is more complicated than it should be... why they didn't just change the char to a short and be done with it I really don't know.

Anyway it should be possible on Solaris 10 to use this fix instead of our "own" implementation of stdio.h, just by adding a couple of lines to gds.cpp
e.g.

#ifdef SOLARIS
#include <stdio_ext.h>
#endif
after including

and then adding the following at around about line 3535 (Firebird 2.0.4)
#ifdef SOLARIS
enable_extended_FILE_stdio (-1, -1);
#endif

Hmm that seems to be a nice simple solution, except that its always more complicated than that.
It seems that this fix is only available from later releases of Solaris 10, so if you have a version of Solaris from
Solaris 10 3/05 through Solaris 10 11/06 you will need to install just 3 patches...
Well no - thats not the case. I had to do the following to get this to work...

Download and install the following patches:

1. 118855-36
2. 118919-21
3. 119255-53
4. 120273-21
5. 123840-04
6. 127756-01
7. 125504-02
8. 122661-08
9. 125548-02
10. 126420-01
11. 126424-03

And then finally in single user mode, followed by a reconfigure reboot:
To boot Solaris x86 in single user mode, requires some fun and games too...

At the GRUB menu, type 'e' before it times out and takes the default boot option.
Scroll to the default multiuser boot line, then type 'e' egain.
This will bring you in to edit mode. Add a ' -s' at the end of the line.
Then type 'b' to boot.

12. 120012-14

Install this patch - then reconfigure

At the GRUB menu, type 'e' before it times out and takes the default boot option.
Scroll to the default multiuser boot line, then type 'e' egain.
This will bring you in to edit mode. Add a ' -r' at the end of the line.
Then type 'b' to boot.

I am not sure that we can ask Solaris users of Firebird if they do not have the fix already installed to go through this, so we will have a look at keeping our own SfIO. However currently that will not compile using gcc 4.2.1 and thats something else we need to try and fix.

Addendum:
sftp is now broken - so you need to add two more patches to the list above

13. 119253-26
14. 124629-07

Thursday, June 5, 2008

MacOSX and stdin,stdout,stderr

I am in the process of porting a friends software to MacOSX. Don't ask what it does yet, all will be revealed soon. The code compiles quite happily on Solaris and Linux, but fails with the following error on MacOSX...

Why?

dbFile.c: In function ‘_setLogfile’:
dbFile.c:243: error: invalid lvalue in assignment
dbFile.c:244: error: invalid lvalue in assignment
dbFile.c: In function ‘fieldFillGen’:

I know what the problem is -
Seems on MacOSX 10.4 and below:
stdin, stdout, and stderr are addresses of a FILE rather than a FILE*
On MacOSX 10.5 they are FILE*

So depending on the OS version you are targeting will determine whether the code compiles or not. The issue goes back to the way FreeBSD originally implemented this in stdio.h which is the base for MacOSX, and it looks like it finally became "correct" i.e. portable and adhering to other implementations in MacOSX 10.5
A quick look at /usr/include/stdio.h confirms this...
the FILE* appears within the ifdef for _DARWIN_UNIX_03

__BEGIN_DECLS
#if __DARWIN_UNIX03
extern FILE *__stdinp;
extern FILE *__stdoutp;
extern FILE *__stderrp;
#else /* !__DARWIN_UNIX03 */
extern FILE __sF[];
#endif /* __DARWIN_UNIX03 */
__END_DECLS


So I need to come up with a simple workaround for what was originally valid code on other operating systems:

stderr = wk;
stdout = wk;

the following seems to work:

dup2(wk->_file ,STDERR_FILENO);
dup2(wk->_file ,STDOUT_FILENO);

Wednesday, June 4, 2008

Standard delete v's Firebird delete

Alex and I have just committed a slew of changes to the code in Firebird 2.1.
This all started when we did the 32 bit builds for Firebird on MacOSX.

We found that in remote we would get asserts trying to create a database. On deeper and further investigation over a period of a couple of weeks we realised that in some cases the standard delete was being called instead of the redefined version in Firebird i.e. it invoked free() instead of
MemoryPool::globalFree(). After some extensive googling we found other people have also run into similar problems.

As a solution it was suggested to use:
1. Use switches -fvisibility=hidden and -fvisibility-inlines-hidden together.
2. Use switch -fvisibility=hidden and __attribute__((always_inline)).

Trying with second combination we found that we sometimes got (depending on the version of gcc) an error:
sorry, unimplemented: inlining failed in call to 'void operator delete(void*) throw ()'.

It seems that without -fvisibility=hidden and __attribute__((always_inline)) gcc is
using non-inline body for operator delete. And is randomly using the wrong body, the one from the standard
operator delete.

The first combination helps to avoid this bug - but any entrypoint, which is to be exported, now has to be marked with __attribute__((visibility="default")).


Monday, April 21, 2008

64bit PPC build for MacOSX

Someone asked me why there wasn't a 64bit version of Firebird for MacOS powerpc... so I tried a build on the powerpc MacOSX box we have...
sw_vers -productVersion = 10.4.11
Build fails when we try to build config_root.cpp.

/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:272: error: typedef 'ExceptionHandlerProcPtr' is initialized (use __typeof__ instead)
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:272: error: 'ExceptionInformation' was not declared in this scope
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:272: error: 'theException' was not declared in this scope
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:306: error: 'ExceptionInformation' was not declared in this scope
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:306: error: 'theException' was not declared in this scope
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:307: error: expected primary-expression before 'userUPP'
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:307: error: initializer expression list treated as compound expression
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/fp.h:1338: error: 'SIGDIGLEN' was not declared in this scope


Some googling explains what the problem is. Seems MacOSX is 10.4 is not 100% 64bit friendly and the carbon framework will not support a full 64bit build until MacOSX 10.5.

We use the carbon framework in a few, but very specific places, mainly to load UDF libraries as bundles and also the fbintl library.

So....

I either need to upgrade our powerpc Mac to MacOSX 10.5 to do the build...
or
Anybody out there got a powerpc Mac running MacOSX 10.5 that I could use to check the build?


Tuesday, April 8, 2008

Editline, Curses and Solaris AMD64

We finally solved the last major problem in the Firebird 2.0x build for Solaris 10 AMD 64bit. Whenever we tried to include editline in the build isql_static would coredump. We tracked the problem down to the following:

Seems there is a problem in the Solaris' native implementation of libcurses, the function tgetstr() returns a bad pointer (non-NULL, but out of the process space). This means that until Sun fix it we can't use libcurses to link with editline on Solaris 64bit AMD.

So we came up with a solution...
We downloaded and built ncurses... (5.5) then copied libncurses.a to gen/firebird/lib along with the normal editline.a
Then adjusted make.defaults to remove references to -lcurses for LINK_LIBS and STATICLINK_LIBS and added -lncurses.

We now have a succesful build that seems to work properly. I hope to put the 64bit Solaris AMD builds up in the firebird prerelease area for testing purposes, then commit the changes made to the code base. Then wen 2.04 is ready to be released will create proper versions for full release.

Hope that somebody out there finds this useful, or at least the builds anyway.

Monday, April 7, 2008

Wrongly Named libfbclient on Solaris

There is a small problem with the currently shipping versions of Firebird 2.0.3 for Solaris.
The problem doesn't affect the general workings of Firebird, but if you try to link with the libfbclient library using your own program, you will run into a problem where you will consistently get the following error:

ld: fatal: library -lfbclient: not found

The problem is due to a misnamed libfbclient.so in /opt/firebird/lib

To correct:
cd /opt/firebird/lib
rm libfbclient.so
ln -s libfbclient.so.2 libfbclient.so

This has already been corrected in CVS



Tuesday, April 1, 2008

64bit Firebird for Solaris 10 AMD64

Looks like having a 64bit build of Firebird on Solaris 10 AMD64 is going to be a non-starter.
Alex and I have done all the hard work, and we have a build (currently without editline, since thats a problem in itself), but when we try and run isql created by the build we get the following:

ld.so.1: ./isql: fatal: relocation error: R_AMD^$_32S:
file /usr/lib/64/libfbembed.so.2: symbol (unknown): vakue 0xfffffd7ffee00000
does not fit

or in debug mode trying to run isql from the build directly:
ld.so.1: isql: fatal: relocation error: R_AMD64_PC32
file /home/pbeach/firebird64/gen/firebird/lib/libfbembed.so.2:
symbol main: value 0x2800155d494 does not fit.

A bit of googling finds this:

Bug ID 5102797
Synopsis AMD64: static relocation in dynamic library fails

If you read the details, seems the bug has been around since 15th Sept 2004
and because of its nature, there are no plans to fix it until Solaris 11.

Addendum: 2nd April 2007
Looks like we have found the problem.
LIB_LINK_OPTIONS in the prefix build file was -G
-G on Solaris should tell the linker to create a shared library
However because we are using gcc, something somewhere was getting confused.
If we pass -fPIC and -shared instead the problem is resolved.

From the gcc manaual:


-shared

Produce a shared object which can then be linked with other objects to form an executable. Not all systems support this option. For predictable results, you must also specify the same set of options that were used to generate code (-fpic, -fPIC, or model suboptions) when you specify this option

On some systems, `gcc -shared' needs to build supplementary stub code for constructors to work. On multi-libbed systems, `gcc -shared' must select the correct support libraries to link against. Failing to supply the correct flags may lead to subtle defects. Supplying them in cases where they are not necessary is innocuous.

Friday, March 28, 2008

FreeAdhocUDF for MacOSX

I have just finished porting FreeAdhocUDF to MacOSX and sent the Makefiles, code, plus 32bit UDF libraries for Firebird 1.5.5 and Firebird 2.x to the FreeAdhocUDF types to be included in the distribution.

Issues that you might need to be aware of.
1. In Firebird 1.5.5 we don't ship libib_util.dylib, and even if we did you wouldn't be able to link to it. It seems that Firebird 1.5.5's installer doesn't copy the library to the Framework. This has now been fixed for the next release. Secondly the library was being built as a bundle. All Firebird UDF's need to be created as bundles to work on MacOSX. You can't link a bundle to a bundle. This has also been fixed.
2. In Firebird 1.5.5 you must remove any extension from the UDF library (e.g dylib) to get it to work. So for example FreeAdhocUDF.dylib needs to be renamed to FreeAdhocUDF. This bug has been fixed in flu.cpp for the next version.

Firebird 2.x does not have any of these problems :-)

In compiling FreeAdhocUDF for MacOSX, it has to be noted that MacOSX has no idea of sysinfo.h, so I had to add a chunk of code to handle the getting of a MAC address, and to get the random_info/randomness for GUIDs. This does seem to work nicely now.

The FreeAdhocUDF libraries I have provided are 32bit intel for Firebird 1.5.5 and Firebird 2.x, if you want to build ppc libraries, or 64bit libraries enjoy. If you have problems, let me know.

On ppc you "might" have to add the following to the Makefile
i.e. change
LDFLAGS=$(CF) $(LIB) -bundle -flat_namespace -undefined suppress to
LDFLAGS=$(CF) $(LIB) -bundle -flat_namespace -undefined suppress /usr/sys/bundle.o

Friday, March 21, 2008

Exceptions, gcc and Solaris 10 AMD 64bit

For a little while now Alex and I have been battling a problem on Solaris 10 64bit AMD. We have been trying to port Firebird 2.0x to 64bit Solaris. However after fixing a few issues, e.g. defining the platform etc, we found that isql_static would core dump on trying to create a database. This is where the fun begins...

The default debugger for gcc shipped with Solaris 10 can't handle 64bit applications. Solution - build your own from a later version. At least this way we can now step through the Firebird code and try to see whats happening..

After some careful debugging sessions we find that Firebird seems to be OK, when you try to create a database, Firebird first trys to open it as an existing database, if it can't find an existing database we throw an internal exception and go on to create it.

The problem is isql_static is occurring when the exception is thrown...

Program received signal SIGSEGV, Segmentation fault.
0x0000000000488385 in __EH_FRAME_BEGIN__ ()

(gdb) bt
#0 0x0000000000488385 in __EH_FRAME_BEGIN__ ()
#1 0xfffffd7ffedacf3c in _Unwind_RaiseException_Body () from /lib/64/libc.so.1
#2 0xfffffd7ffedad129 in _Unwind_RaiseException () from /lib/64/libc.so.1
#3 0xfffffd7ffef3c71e in __cxa_throw (obj=0x1, tinfo=0x1, dest=0x474e5543432b2b00)
at /builds/sfw10-gate/usr/src/cmd/gcc/gcc-3.4.3/libstdc++-v3/libsupc++/eh_throw.cc:75
#4 0x000000000056510b in Firebird::status_exception::raise (status_vector=0x8d19e0)
at ../src/common/fb_exception.cpp:197
#5 0x0000000000694137 in ERR_punt () at ../src/jrd/err.cpp:562
#6 0x0000000000693ced in ERR_post (status=335544344) at ../src/jrd/err.cpp:441
#7 0x00000000005c66d4 in PIO_open (dbb=0xfffffd7ffec29050, string=@0xfffffd7fffdec9e0,
trace_flag=false, connection=0x0, file_name=@0xfffffd7fffdeca20, share_delete=false)
at ../src/jrd/os/posix/unix.cpp:646

So Firebird looks OK, i.e. its doing what it should do, however Solaris is causing a crash dump on the exception.

Some more detective work uncovered the following:

The problem is caused by the use of the -lc flag which was explicitly added to linker command line by autoconf.

Because we always use g++/gcc as linker, there is no need for us to need add in libc explicitly - it's done by compiler itself when it invokes ld. But autoconf seems to be doing it anyway.

When it is added to the command line, libc (Solaris native library) then happens to be added to ld's command line before libgcc_s where we would find gcc's own exception support library. If this happens any call to functions that have the same name, that happen to be present in both libraries, will go to libc, not libgcc_s.

There is at least one such function - Unwind_RaiseException(), which is called by g++ generated
code when exception is thrown. But instead of libgcc_s::Unwind_RaiseException() libc::Unwind_RaiseException() is called, but not with the parameters the Solaris native function expects - cue core dump.

Solution:

Don't use -lc

Thursday, March 13, 2008

Where is libfbclient.dylib on MacOSX?

There was a post to the Firebird Support list, asking how you link Qt to the libfbclient.dylib on MacOSX Firebird SuperServer. The problem being, that libfbclient.dylib is not "shipped" as part of the MacOSX framework... heres the detailed answer on how it works.

Firebird on MacOSX is delivered as a framework,

For more information on a frameworks look on : http://developer.apple.com

In a nutshell - "A framework is a hierarchical directory that encapsulates shared resources, such as a dynamic shared library, nib files, image files, localized strings, header files, and reference documentation in a single package. Multiple applications can use all of these resources simultaneously. The system loads them into memory as needed and shares the one copy of the resource among all applications whenever possible."

This is the "best" way to deliver an application on MacOSX.

However to do this the main dylibs (shared libraries) that run Firebird become part of the framework itself.

The main dylib that drives Classic is libfbembed, for SuperServer its libfbclient.

With Classic we ship both libfbembed and libfbclient, libfbclient as a separate dylib can be found in the libraries directory.

for SuperServer we ship libfbclient

However libfbembed (Classic) and libfbclient (SuperServer) are not directly obvious as shared libraries, i.e. if you look for them by that name with a dylib extension you are not going to find them. However if you took a close look at postfix.darwin in the Firebird source builds/posix (this is the code that sets up the framework) you will find that they become Firebird.framework/Firebird which is a link to Firebird.framework/Versions/Current/Firebird.

Where
Firebird.framework/Firebird is libfbembed.dylib for Classic and
Firebird.framework/Firebird is libfbclient.dylib for SuperServer



Tuesday, March 11, 2008

Building GCC 4.2.1 On Solaris

Alex and I are currently trying to implement a 64bit version of Firebird 2.x for Solaris 10, and we are running into a number of problems. However they shouldn't be insurmountable. Every port has its problems one way or another, but this one is proving to be particularly annoying....

The default version of gcc shipped with Solaris 10 is 3.4.3 and is found in /usr/sfw/bin
The current available version of gcc (released is) 4.3
So the version Sun makes available dates from 2005...
One assumes that Sun has no interest in letting Solaris users use more modern versions of gcc and would rather we use their own compiler.
Hmm so much for Sun and open source...

But to properly compile on Solaris 10 with 64bit see notes from gcc.gnu.org

"
i?86-*-solaris2.10
Use this for Solaris 10 or later on x86 and x86-64 systems. This configuration is supported by GCC 4.0 and later versions only.

It is recommended that you configure GCC to use the GNU assembler in /usr/sfw/bin/gas but the Sun linker, using the options --with-gnu-as --with-as=/usr/sfw/bin/gas --without-gnu-ld --with-ld=/usr/ccs/bin/ld"

So gcc 4.0 is the lowest version you can use for cross-compiling 32bit and 64bit applications on Solaris 10...

So we have to build our own gcc to do the firebird compilation, here's how you do it....
1. Download a copy of the gcc source (e.g. gcc-4.1.2.tar.gz) from gcc.gnu.org
2. Install gnu tar, solaris tar has problems extracting the source from the gcc tar file.
3. You will probably need to install libintl too (you can get a pre-built version of this froim www.sunfreeware.com)
4. gunzip gcc-4.1.2.tar.gz
5. /usr/local/bin/tar -xvf gcc-4.1.2.tar
6. cd gcc-4.1.2
7. mkdir objdir
8. cd objdir
9. ../configure --with-gnu-as --with-as=/usr/sfw/bin/gas --without-gnu-ld --with-ld=/usr/ccs/bin/ld --enable-shared --enable-languages=c,c++
10. gmake
11. as root - make install

Your new version of gcc will be installed in /usr/local/bin

Friday, February 29, 2008

Symbol Variants - MacOSX

Up until now it has been impossible to run kinterbasdb against SuperServer builds of Firebird for MacOSX on Tiger (10.4). When you try and import kinterbasdb into Python the following error occurs:
Symbol not found fputs$UNIX2003
This is a bit of a problem.
Reading around the issue indicates that symbol usage between MacOSX 10.4 and 10.5 has changed because of the introduction of
UNIX cconformance.
This article on developer.apple.com explains why we might see UNIX2003 symbols.
Basically anything compiled on 10.5 will use the new conforming UNIX2003 sysmbols where appropriate.
However we shouldn't be seeing these symbols if we compile everything with
--mmacosx-version-min=10.4 and/or set MACOSX_DEPLOYMENT_TARGET=10.4
So why when we try and run kinterbasdb on 10.4 we have a problem?
A nm -a libfbclient.dylib shows that there are 4 UNIX2003 symbols in the library...
and I had no idea where they were coming from.
It seems that we have a link problem, and the only way to make sure that they do not get pulled in to the libfbclient library is to insist that the linker links against the 10.4 sdk.
A small change to prefix.darwin_i386 seems to solve the problem
Add LD_FLAGS+=-isysroot /Developer/SDKs/MacOSX10.4u.sdk
Why the linker ignores the flag/environment variable is a mystery to solve another day.

Thursday, February 21, 2008

Firebird and MacOSX Uninstalls

MacOSX doesn't have an uninstall utility.
I use the following script to clean up Firebird installs on Leopard.
It should work on Tiger as well.
Would it make sense to include this in the Firebird package, along with a similar script for earlier versions of MacOSX (pre 10.4 for the powerpc builds)

#!/bin/sh
echo "Clean Services"
echo "Clean User"
dscl localhost -delete /Local/Default/Users/firebird
echo "Clean Group"
dscl localhost -delete /Local/Default/Groups/firebird
if [ -f "/Library/StartupItems/Firebird" ]; then
echo "Remove SuperServer StartupItem"
rm -fr /Library/StartupItems/Firebird
fi
if [ -f "/Library/LaunchDaemons/org.firebird.gds.plist" ]; then
echo "Remove Launchd"
launchctl unload /Library/LaunchDaemons/org.firebird.gds.plist
rm /Library/LaunchDaemons/org.firebird.gds.plist
fi
echo "Remove Framework"
rm -fr /Library/Frameworks/Firebird.framework
echo "Remove Receipt"
rm -fr /Library/Receipts/Firebird*.pkg

#Add the following for Firebird V2.5
echo "Remove /tmp/firebird"
rm -fr /tmp/firebird

Tuesday, February 19, 2008

Python, kinterbasdb and 64bit MacOSX 10.5

Now we have 64bit builds of Firebird 2.1 for MacOSX 10.5, we have issues trying to test them properly using the Firebird QMTest QA suite. This QA suite relies on Python and kinterbasdb. Currently Philippe Makowski has been able to run the tests from a 64bit Linux client. But is unable to run the tests locally. Well currently the python framework that comes with MacOSX 10.5 is only 32bit and this forces a 32bit build of kinterbasdb, since the build environment of kinterbasdb uses distutils to use the same compile switches as python. So I thought I would have a look at the issue and see if I could build a 64bit version of kinterbasdb for MacOSX 10.5.

This is how I did it...

1. Download the source distribution of Python
2. Edit configure.in and change line 817, so it looks like this
BASECFLAGS="$BASECFLAGS -Wno-long-double -no-cpp-precomp -mno-fused-madd -arch x86_64"
3. Make sure that (just in case)
a, CFLAGS, CXXFLAGS and LDFLAGS are exported with -arch x86_64
e.g. export CFLAGS='-arch x86_64'
b, export MACOSX_DEPLOYMENT_TARGET=10.5
4. autoconf
5. ./configure --disable-toolbox-glue
This disables Carbon as its only 32bit and you will get compile errors if you don't
We are not building a framework, or a universal binary just a simple standalone version of 64bit python. So do not enable those options.
6. make
You may get a warning/errors possibly in some of the extensions e.g.
ld warning in /System/Library/Frameworks/TK.framework/Tk missing required architecture x86_64 in file.
I guess you would have to download the relevant code and rebuild these as 64bit if you need them. But since we are only interested in kinterbasdb I guess I can afford to ignore it.
A check on the built python.exe shows that it is Mach-O 64 bit executable.

Now we need to build kinterbas db using this python.exe
1. Download kinterbasdb from sourceforge
-> 2. Replace in the kinterbasdb source the __init__.py with this one :
https://firebird.svn.sourceforge.net/svnroot/firebird/qa/trunk/kinterbasdb/__init__.py
Note I forgot to add this step when I wrote the original instructions.
3. Make sure you have the 64bit Firebird 2.1 for MacOSX installed
4. Make sure you are running as root
su and log in
export MACOSX_DEPLOYMENT_TARGET=10.5
export CFLAGS, CXXFLAGS and LDFLAGS ='-arch x86_64'
The install option will want to put the kinterbasdb in /usr/local/lib/python2.5
Although I suspect you can change this by using a different --prefix when you build python
5. pathtojustbuilt64bitpython/python.exe setup.py install
or
pathtojustbuilt64bitpython/python.exe setup.py build
if you just want to build it.
6. You can check what you have installed by
file /usr/local/lib/python2.5/site-packages/kinterbasdb/_kinterbasdb.so
Mach-O 64bit bundle x86_64


Now we can test it...
create a database using isql
pathtojustbuilt64bitpython/python.exe
>>> import kinterbasdb;kinterbasdb.init(type_conv=200)
Don't use mx.DateTime use the default python DateTime
>>> con = kinterbasdb.connect(dsn='localhost:/pathtodd/test.fdb', user='sysdba', password='masterkey')

It works ....


Tuesday, February 12, 2008

Testing and Debugging the Service Manager on MacOSX

For Classic - SuperServer is easy.

PATH=$PATH:/Library/Frameworks/Firebird.framework/Resources/bin
export PATH

In one terminal session:
gsec -user xxx -password -xxx -database localhost:/Library/Frameworks/Firebird.framework/Resources/English.lproj/var/security2.fdb
This will start a remote connection to gsec via the service manager...
ps -eaf | grep fb
and look for the running fb_inet_server

In the other
cd /Library/Frameworks/Firebird.framework/Resources/bin
gdb ./fb_inet_server --pid=
Set the pid to the running fb_inet_server
gdb> break svc.cpp:2461
set an appropriate break point - in this case when vfork starts
gdb> continue

In the original session
gsec> display

In the gdb session the debugger should break when vfork is called in svc.cpp

Now do what you need to - to walk through the code.

If all is working properly, try doing a backup using the service manager...
gbak -v -b -user xxxx -password xxxx /Users/pbeach/databases/test.fdb /Users/pbeach/databases/test.fbk -se localhost:service_mgr

Friday, February 8, 2008

Adding a User to a Group using dscl

How to add a user to the Firebird group on MacOS using dscl

dscl localhost append /Local/Default/Groups/firebird GroupMembership pbeach
or
dscl . append Groups/firebird GroupMembership pbeach

Sunday, February 3, 2008

Nine Below Zero

Non Technical for a change:

2nd Feb 2008 + Cafe Francaise@Chinon + Nine Below Zero
* (Rhythm + Blues) + 150 people = Superb gig.

Friday, January 18, 2008

64bit Firebird for MacOS 10.5 (Leopard)

Today I put the finishing touches to a native 64bit port of Firebird for MacOS 10.5 (Leopard).
This required the following changes to the Firebird 2.1 code base (B2_1_Release). For anybody else considering adding a new platform to Firebird then you may find this useful.

Changes:
1. src/jrd/common.h (line 289)
#ifdef __x86_64__
#define DARWIN64
#define IMPLEMENTATION isc_info_db_impl_darwin_x64 /* 73 */
#endif

2. src/jrd/inf_pub.h (line 203)
add
isc_info_db_impl_darwin_x64 = 73,

3. src/jrd/license.h (line 135)
amend
#ifdef DARWIN
#if defined(i386)
#define FB_PLATFORM "UI" /* Darwin/Intel */
to
#if defined(i386) || defined (__x86_64__)

4. src/jrd/pag.cpp (line 155)
insert (last)
static const int CLASS_DARWIN_X64 = 31; // Darwin x64
amend (line 158)
static const int CLASS_MAX = CLASS_DARWIN_X64;
since this is the last one added

5. src/jrd/pag.cpp (line 245)
insert (last)
archLittleEndian // CLASS_DARWIN_X64
make sure you add a , to the class above...

6. src/jrd/pag.cpp (line 322)
Pick up the right class for the right architecture..
add
#ifdef DARWIN64
const SSHORT CLASS = CLASS_DARWIN_X64
#endif

7. src/remote/protocol.h (line 150)
add a new architecture
arch_darwin_64 = 41,
and update arch_max
arch_max = 42

8. src/remote/remote_def.h (line 72)
add a new architecture
#elif defined(DARWIN64)
const P_ARCH ARCHITECTURE = arch_darwin_64;

9. src/remote/xdr.cpp
Make sure SWOP DOUBLE is defined correctly
by appending
|| defined(DARWIN64)
to the intel crowd.

10. src/jrd/utl.cpp (line 218)
add
"Firebird/Darwin/Intel64" /* 73 */

11. src/common/classes/fb_atomic.h (line 82)
add || defined (__x86_64__)

12. configure.in (line 62)
In the target darwin section add
dnl CPU_TYPE=x86_64
This can be uncommented when you do a 64bit build


Now read the notes in builds/posix/prefix.darwin_x86_64

1. edit configure.in so that MAKEFILE_PREFIX=darwin_x86_64
2. Uncomment the CPU_TYPE
3. edit extern/icu/source/config/mh-darwin to set the right 64bit flags (-arch x86_64) for LD_FLAGS
for CFLAGS, CXXFLAGS
4. export CFLAGS='-arch x86_64'
5. export CXXFLAGS='-arch x86_64'

Then check and make sure that
MACOSX_DEPLOYMENT_TARGET=10.5
and export that for good measure.

Build should be good to go. Welcome to native 64bit Firebird on MacOS

Wednesday, January 9, 2008

depot Files and HP-UX

I have just finished writing a depot install for Firebird Classic 2.0x on HPUX . A depot install is the HP equivalent of a Linux rpm or Solaris package. The operating system install uninstall utilities take command of looking after the files. Although in more complex areas you still have to have the relevant postinstall and postremove scripts to handle things like links etc.
To allow a user other than root to use SD-UX you need to give yourself access to it e.g.
swacl -m user:pbeach:crwit -l host
To create a depot install you need to write a .psf file (the Firebird specific one will be found in builds/install/arch-specific/hpux/classic/firebirdCS.psf), you then

test it using:
swpackage -vv -p ./firebirdCS.psf

create it using:
swpackage -d "| gzip -c > /firebird2/gen/firebird.depot.gz" -x media_type=tape -vv -s firebirdCS.psf

install it using:
swinstall -s /firebird.depot.gz

list installed packages using
swlist

remove it using
swremove Firebird

Open Source Code Contains Security Holes

The following was sent to Charles Babcock at Information week in reply to an article entitled:

Open Source Code Contains Security Holes

As a developer and administrator of the Firebird Project I completely reject the statement you made in the above article.

"The somewhat moribund Firebird project, for example, is listed with 195 identified defects, of which it has verified zero and fixed zero. The active Firefox browser project, on the other hand,
has fixed 370 bugs, verified 56 and faces another 246 to verify and fix."

The Firebird project is in fact incredibly active - perhaps a look at this chart on our bug tracker might give you a clue.

http://tinyurl.com/yt5pgl

Firstly the Firebird project reviewed the Coverity results almost immediately they were published and found that the report isn't actually related to the Firebird engine. This URL shows our appropriate comments from the 7th March 2006:
http://www.firebirdnews.org/?p=180

Also more comments from Claudio on the 26th March 2006:
http://www.firebirdnews.org/?p=243

Secondly in a more detailed reply to the actual "PR" issue raised by David Maxwell, open source strategist for Coverity. If you had asked about this before printing the article you could have put some facts straight.

Nearly all of the 195 identified defects are in fact actually within an external piece of code we use for character sets and collation sequences ICU

http://www-306.ibm.com/software/globalization/icu/index.jsp

"The International Component for Unicode (ICU) is a mature, portable set of C/C++ and Java libraries for Unicode support, software internationalization (I18N) and globalization (G11N),
giving applications the same results on all platforms."

A open source project maintained by IBM. I will admit that we are using an older version of ICU (3.0) than is currently available and we will be upgrading to a newer version in the near future.
But this is not something that is a trivial exercise, as it means that any database using a different version of ICU would be incompatible with the version we ship. We plan to upgrade ICU
in Firebird version 2.5

Other defects reported are one in
usr/include/c++/4.0.2/i386-redhat-linux/bits/gthr-default.h
Not our problem either....

And there are four defects in firebird2/src/gpre/pretty.cpp a piece of old code used with a pre-compiler (gpre) to make BLR look good. BLR (Binary Language Representation),
Firebird's internal compiled language. This doesn't affect the Firebird server at all.

I would like you to print a correction or at least acknowledge the innacuracy of the article as regards Firebird.

Regards
Paul Beach

Monday, January 7, 2008

Firebird 2.03 on HPUX 11

I have just released a classic version of Firebird V2.03 for HPUX 11. This is a 64bit version for PA-RISC 2.0. The current install mechanism as per Firebird 1.5.x uses a tar file and scripts to install firebird into /opt/firebird etc. I think the time has now come to try and write a depot file so that we can use the HPUX installer to install firebird. Initial analysis indicates that I need to write a .psf file that contains the relevant instructions, and that I can test it using the following command:
swpackage -p -vv -s ./firebirdCS.psf -d /sd/firebird
Where:
-p = preview mode (don't do anything - just test the psf file)
-vv = very verbose
-s = source psf file
-d = destination