Top Banner

ZAppDB Installation and Programming


The following is developer support documentation. This documentation reflects the latest information available. We intend to expand and improve this document over time, so you may want to mark this page and return often if you are a developer using ZAppDB.

Content:

Platform Installation Options

ZAppDB Core is a Java application designed to run as an application or as a service on a wide variety of support OS's. You need to use the proper installation file for your specific operating system. A generic version without the java service wrapper is available for currently unsupported operating systems. Currently supported OS's include the following:

  • Linux (64 bit)
  • Windows
  • Generic, no wrapper services.

Using the Java Service Wrapper allows you to start, stop, and restart ZAppDB Core. The wrapper allows you to run ZAppDB Core as a daemon process. The wrapper also provides nice logging services for System messages.Pre configuration

With this new version ZAppDB Core analyses how data is accessed, so that ZAppDB Core can configure itself to provide data requests as quickly as possible, while using memory efficiently. As each new request is analyzed, and entry is stored in ZAppDBPreCfgInfo.cfg.

Pre configuration (ZAppDBPreCfgInfo.cfg) entries take the following form:

 T:DEMO2~I:[ID]~F:[FNAME, LNAME]~C:51296~S:2 These entries can be deleted before startup. You can also add
		  definitions in advance if you follow the precise form as shown.
 S: is for internal use only. If you add a new record then you can put '0' in place of the '2'.
 C: is for internal use only, but refers to the database size when the original search was requested. 
          If you are manually creating an entry you can replace '51296' with '0'.
 T: is the table alias.
 I: and F: are the column lists used for the index and the fields you are returning respectively. 
          The form is [column1Name, column2Name, ...]

As a developer you can use precfg information to determine whether you can optimize queries to reduce memory requirements. If two or three queries a identical except that there are slightly different columns in the fields specification, you can consolidate queries into a single query form. This will significantly reduce the amount of memory you need for caching, and reduce the amount of time required to load data from the SQL server. Studying the precfg file can be very helpful.


ZAppDB Core Configuration Info

Here is an example ZAppDB.cfg file:

#ZAppDBDataServer Configuration File
#Fri May 5 12:06:11 MDT 2000
SecurityPrefix=NoNe            ; This token is used as a simple request authentication
Threads=100                    ; The number of concurrent requests you want to handle.  
                               ; 100-500 is a typical range of values
IPAddress=10.10.10.10          ; This is the IP address you want to bind ZAppDB to.  
Port=50104                     ; This is the port you want to use
ZipTmpFilesPath=zipTmpFiles    ; A temporary directory for zip temporary files 
FilesRoot=filesRoot            ; The root directory for file transfers
RemoteControlPort=9987         ; The IP address of the server, and this port defined remote access (0 disables)
RemoteSecurityPrefix=NoNe      ; This token is used as a simple remote request authentication
WriteLockTimeoutMinutes        ; This establishes the max length of time in minutes that a record can be locked

The correct number of threads is variable and application dependent. Each thread uses memory, and as the number of threads increase there is a reduction in efficiency. If you have a lot of memory, and the ratio of reads to writes is very high then increasing the number of threads might be a good idea. We have tested with 1000 threads, but 100 to 500 threads will be best in most cases. Once your server has reached maximum utilization, increasing threads will not give you much of an improvement in throughput, but you will be able to service more concurrent requests. Too many threads can actually degrade system performance.

As you gain experience using ZAppDB Core and after monitoring system parameters with ZAppDBRC you can get a 'feel' for adjusting the number of threads. Requests are handles so quickly that a hundred threads will likely be a lot more than you need. A hundred concurrent requests can be handled in 1 ms, so you don't need to be super aggressive unless your doing something unusual in your application.

Cache memory requirements are very application sensitive.

Maximum tested heap is 1.6GB on 32 bit systems and >6GB on 64 bit operating systems. Be sure to use the 64 bit version of java (if you have a 64 bit server) for the server. If you specify too large a heap, java will not create a JVM. The heap size is specified in the wrapper.conf file in conf directory of your working directory. If heap is too small performance is hampered, but will still give increased performance. But for maximum benefit use a large heap.

If you watch ZAppDBRC you can see how the heap is being utilized over time. Database Configuration

Configuring a database is fairly simple. The first thing to do is to assign a name. This database name is a name that will be used to identify this specific database engine.

Tables Configuration

The table configuration tool works in a very similar way to the database configuration tool. Editing options are the same.

Tables are a subset of the database. The 'Table Alias Name' is the name you wish to use in referring to this specific table. The 'Table Reference Name' is the table name reference required to access the table. In the example on the left, the table needs to be referenced as MATEX.DBUSER but we can just use DBUSER to refer to the table.

The Unique Field Name is the name of a unique (no duplicates) field that ZAppDB can use to identify a specific record. All tables are required to have a numeric column that will used as a unique id field. This is required for proper operation of ZAppDB. The uniqueid column can be named as you prefer. The "Enable Auto-Increment checkbox is no longer an option.

After the fields are filled in, be sure to select 'Save Changes' so that the changes can be made to the configuration file.

EXIT AND RESTART ZAppDB so that these changes will be in effect. In this version a restart is required.


Sample Request Structure Format

The request structure is as simple as it can be. More exotic searches can use stored procedures, aliases, etc, but for what we need this format will do the job.

Any extra intelligence can (and will) be applied on the server side. This will keep the request easy to implement.

The following request will be used as an example:

AccessKey_$%#@!##             ; The first line is actually a key, required to gain access to ZAppDB services
                              ; The next symbol will be W: for a write operation, or R: for a read operation.  
                              ; These are mutually exclusive terms.  Only one or the other may be used.R: or
                              ; R: is a read operation 
                              ; RL: is a read that locks the record
                              ; [W:I, W:U, W:D, W:CI] ; The initial read request (R:) has no ID assigned
                              ; W:D signals a Delete write operation
                              ; W:I signals an Insert write operation
                              ; W:U signals an Update write operation         
                              ; W:CI signals a Cached Insert write operation
                              ; WL: same as a normal write but clears a lock
                              ; WLC: (after v1.15) signals a Write Lock Clear
U:GenericServices, Inc.       ; Company info for the log.  In case there is a user with issues.
T:ADAPTERS                    ; T=type I=index F=FieldName  Type is the normally the database name.
I:COMPANYNAME;Mag             ; This lines starts with I: indicating that this is an index key
                              ; The field to use as an index is COMPANYNAME.
                              ; If there is a ':' the text following is a complete key value
                              ; A ';' indicates that the text following is a partial key
I:BUSTYPE;ISA                 ; This is a secondary index key, with "ISA" as a partial key
F:PRODUCTID                   ; This is a return field, and is returned as the first field
F:DESCRIPTION                 ; This field's content will sit in the second position
                              ; FRC: (after v1.15) specifies a Field Relative Change request which is valid for
                              ; for integer or real values. This is ideal for incrementing or decrementing values.
O:0                           ; Offset into the list  
                              ; NOTE: O:R:0 means that the offset is relative to current position information
                              ; which was supplied as part of the R: token listed above.  If the request returns a 
                              ; 'no records found' error the most likely , then the request must be made with a 
                              ; specific offset specified here (i.e. O:342) and when the request is returned new 
                              ; extended position information that is valid will be returned. The next request
                              ; then can be relative.
N:50	                        ; Number of lines (records) to return 
                              ; Offset and Number can be used to implement extremely fast 
                              ; list boxes in real time. Ask if you want to know more...

Each line of the request is terminated with a '\n'. The last line of the request always ends with "\n\n".

All requests are in clear text. There are no empty lines. All transmitted requests will have every line filled. If you are using the ZAppDBClient bean and also using contexts, the results are sent in encrypted form, which the bean unencrypts the data for you.

Any field can be used as an index.

All searches are case insensitive, which makes things easier.


The "R:" line can be used to search an existing cached data object. This feature is only useful for block data transactions only! This extra information is used for record style navigation through your data. Partial searches and exact searches do not use this information at this time.

If you have already made a request with a large number of available records you can use this number (that is returned from every request) to get another chunk of data much faster. If you do not specify the extra 'R:' information, the server will look to see if a search with the same parameters has been made and is in cache. By using the extended information, you start right where you left off. which dramatically decreases response time for relational navigation through the data set.

A typical example might look like this: R:3:5:20:2115150400 The first 3 parameters (3:5:20) give the position of the last returned row of data. Future accesses can the start where the previous request left off. The last portion of the string is the timestamp reflecting server start time. This string is in the following format "ssmmHHddMMyy". This portion is critical because the assigned object numbers change when the ZAppDB is restarted. Using this information, a check is made by ZAppDB to verify that the reference information is still valid.

The "T:" references the alias you defined when you configured ZAppDB and refers to a database table name.

The "N:" should contain the number of records you want back. The reply will contain how many matching records are available. If you want more than you requested you can set your offset and request the difference between what you already got and what is there and get the rest. You can use the search id to get what you want immediately from cache. N:All will return all matching records.

The text of this request is order specific. The data returned will also be sorted by the order of the fields automatically. In this example the product Id's would be in ascending order.



Reply Format

The following is an example of the reply you would get from the server:

ZAppDB_v1.15          ; This tells the client what syntax is being used for the reply
R:43:100:40:2115150400   ; As you can see positional data was returned from the request
ERRMSG:                  ; A string that is an error message, only when an error occurs
N:1                      ; 1 item or record is being returned
A:1                      ; There is only 1 record available

17542,CNE2000.LAN,14250,11/25/97 ;The data returned is in csv format and looks like this

All data is returned as one CSV record per line and is in clear text.

If a request was made properly the only time you would get an error returned is when there is no match for the request. If there is an error, an error string will indicate the nature of the error.

There is a blank line between header info and data, so you can know where the header
ends and the data begins. There is no returned data on a failed request.


Dates

Dates will be handled in a standardized format. ZAppDB will convert the date properly to make the request, so that you can always search for dates by the same mechanism. Data returned will be in the same form as the request. You may want to reformat the reply to suit your application, but consistency simplifies things.

Additionally, partial requests are automatically in the proper order of precedence. The dashes are important. Slashes will not work.

I would recommend that all date index requests should be partial. In a case where tenths of a second are used you could fail a search. If you can be sure that this is NOT the case then go ahead and use an exact key if you want. It's up to you....

The date format is YYYY-MM-DD HH:MM:SS (24 Hrs)

The date records used in Oracle contain more than just the dates, but also include hours, minutes, and seconds. If you don't want seconds, you can easily crop that information from the returned string. There will be cases where seconds could be important, so they are there if you need them.

If there is a case where this data format will not work, please let me know. We can always extend this spec. We want to keep this protocol simple as possible.

The database field name must include the string "DATE". For now this is how we will identify whether the field is indeed a data field. I have a better solution but I don't want to invest the time required (for now) to implement it.



Examples

Request

AccessKey_$%#@!##                   ; The access key
R:                                  ; Initial request has no ID assigned
U:GenericServices, Inc.             ; Company info for the log
T:ADAPTERS                          ; T=type I=index
I:COMPANYNAME;Mag                   ; We want lancards by company w/srch string
I:BUSTYPE;ISA                       ; This is a good Idea not currently using
F:PRODUCTID                         ; This sets the field order for the
F:DESCRIPTION                       ; request
O:0                                 ; Offset into the list
N:50                                ; Number of lines to return

Response

ZAppDB_v1.15
R:43:121:11:2115150400
N:5
A:5

14000,MAG200 Warp Core Adapter
271,EXOS 205
273,EXOS 215
272,EXOS 215T
264,EtherLink (3C501)

______________________________________________________________________________

Request

AccessKey_$%#@!##                   ; The access key
R:                                  ; Initial request has no ID assigned
U:GenericServices, Inc.             ; Company info for the log
T:CPUS                              ; T=type I=index
I:COMPANYNAME;Intel                 ; Starts with partial key query
F:PRODUCTID                         ; This sets the field order for the 
F:DESCRIPTION                       ;
O:0                                 ; Offset into the list
N:50                                ; Number of lines to return

Response

ZAppDB_v1.15
R:42:111:11:2115150400
N:11
A:11

14406,Pentium (tm) 266
14407,Pentium (tm) 100
14408,Pentium (tm) 200
14410,Pentium (tm) 120
14411,Pentium (tm) 133
14412,Pentium (tm) 166
14413,Pentium (tm) 150
23654,Pentium Pro (tm) 200
16852,Pentium(r) II 233
22363,Pentium(r) II 300
23291,Pentium(r) II 200

____________________________________________________________________________

Request

AccessKey_$%#@!##                   ; The access key
R:                                  ; Initial request has no ID assigned
U:GenericServices, Inc.             ; Company info for the log
T:BUSTYPES                          ; T=type I=index 
F:PRODUCTID                         ; This sets the field order for the 
F:DESCRIPTION                       ;
O:0                                 ; Offset into the list
N:50                                ; Number of lines to return

Response

ZAppDB_v1.15
R:43:215:23:2115150400
N:23
A:23

5139,32-Bit PCI Hot Plug
5138,64-bit PCI Hot Plug
2436,AGP
4657,AGP II
1413,Apple Nu-Bus
3958,Bus Expansion Slot
4258,CPU-Embedded
5082,CardBus
78,EISA
125,EISA/PCI
199,EISA/VL
3796,Embedded
3956,Expansion Slot
23,ISA
94,ISA/PCI
26,ISA/VL
794,ISA/VL/PCI
402,ISA16
412,ISA8
130,MCA
1410,MCA16
1411,MCA24
1412,MCA32

____________________________________________________________________________

Request

AccessKey_$%#@!##                   ; The access key 
R:                                  ; Initial request has no ID assigned
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type I=index 
I:COMPANYNAME;Seagate               ; 
F:PRODUCTID                         ; This sets the field order for the 
F:DESCRIPTION                       ;
O:0                                 ; Offset into the list
N:50                                ; Number of lines to return

Response

ZAppDB_v1.15                        ;
R:42:111:2:2115150400               ;
N:2                                 ;
A:2                                 ;

22943,3232A
22626,34371WC

Writing Data

The "W:" is an alternate choice for "R:" where writing changes to a table is required. Immediately after the colon the specific write operation is specified. U for update, I for insert, and U for update. Unlike a read request, the indexes are not specified. Only the fields are described, and after a colon is the specified data. (i.e. CUSTOMER:Thompson) In the case of a delete operation, the key field is specified in the same way. (i.e. CUST_ID:3442) Later in this document a specific example will be shown.

First Example "Delete"

Request

AccessKey_$%#@!##                   ; The access key 
W:D                                 ; A write operation. In this case DELETE
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type (table alias)
I:PRODUCTID:310                     ; This specifies the specific record to be deleted 

Response

Deleted OK                          ;

Second Example "Insert"

Request

AccessKey_$%#@!##                   ; The access key 
W:I                                 ; A write operation. In this case INSERT
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type (table alias)
F:PRODNAME:WIDGET                   ; Product Name 

Response

Inserted OK                         ;
W:ID:564                            ; One record inserted, new autoincremented entry 564

Third Example "Update"

Request

AccessKey_$%#@!##                   ; The access key 
W:U                                 ; A write operation. In this case UPDATE
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type (table alias)
I:PRODUCTID:54663                   ; This is the id to be modified 
F:PRODNAME:WAM WIDGET               ; Product Name 

Response

Updated OK                          ;

Record Locking

Since ZAppDB is designed for internet remote-enabled database applications, locking has been integrated with a slightly different focus than most traditional database applications.

There are currently three different record locking options:

  • Read after locking.
  • Write and clear the lock.
  • Clear the lock

A background process clears locks after the lock exceeds the user specified timeout. The ZAppDB administrator should specify a reasonable value and the developer of the application should take into consideration the fact that the locks only last for a finite time, so that locking errors can be dealt with properly.

RL: is in all respects the same as the R: the only difference is that the record is locked and then read.

WL: is in all respects the same as the W: the only difference is that after a successful write the record is unlocked.

WLC: is a new tag that clears the lock on the record.

The request for WLC: looks like this:

AccessKey_$%#@!##                   ; The access key 
WLC:                                ; A write lock clear operation. 
U:GenericServices, Inc.             ; Company/User info for the log
T:STORAGEDEVICES                    ; T=type (table alias)
I:PRODUCTID:310                     ; This unique id specifies the specific record cleared 

Sample Java Source

The Java connector that comes with ZAppDB makes requests easier and offers additional capabilities, so we highly recommend using the connector where possible. However, other languages can leverage ZAppDB by making direct queries. ZAppDB should be kept unexposed to the web directly as a general practice. It can be done if you develop a security wrapper for access, but if not done correctly you could expose your data to bad actors.

This code snippet illustrates how specifically a request can be made from ZAppDB directly without the use of a connector. This code could easily be used on any non-Java platform that supports IP.

 //**************************************************************
	public final void sendRequest() 
	{
	   String s = null;
      	   
	   try
	   {
         sock = new Socket(ipAddress, portNum);
         streamout  = sock.getOutputStream();
         
         streamin   = sock.getInputStream();
	      
         BufferedReader in = new BufferedReader( new InputStreamReader(streamin) );
	      
         String outstr = 
                     "AccessKey_$%#@!##\n"+
                     "R:\n"+
                     "U:Larry\n"+
                     "T:PRODUCTS\n"+
                     "I:PRODUCT_KEY:7831\n"+
                     "I:COMPANY_KEY:300\n"+
                     "I:PRODUCT_NAME;Pob\n"+
                     "F:PRODUCT_TYPE_KEY\n"+
                     "F:COMPANY_KEY\n"+
                     "F:PRODUCT_NAME\n"+
                     "O:0\n"+
                     "N:1000\n\n";
	                  
         byte[] baray = new byte[outstr.length()]; 
         baray = outstr.getBytes();
         streamout.write( baray, 0, baray.length );
	      
         // in the test client a 'resultList' box displayed the results of the search
         resultList.add("DATA_RCVD_FOLLOWS..");   
         resultList.add("-------------------------------------------");   
         resultList.add(" ");   
	         
	      try
	      {
	         s = in.readLine();
	         while( s != null )
	         {
	            resultList.add(s);   
	            s = in.readLine();
	         }
	      }
	      catch( IOException ioe )
	      { 
	         System.out.println(ioe); 
	      }
	   }
	   catch(Exception e)
	   { 
	      System.out.println(e); 
	   }
	}

          

In this example you can see both how to make a query, and how to get the results. The first blank line in the result marks the beginning of the data being returned. Please look at the Reply section and the examples for further information.


Cached Writes

Some write operation do not need to be completed before continuing to the next step. Cached Writes allows these transactions to be queued safely, until database activity is reduced. Then the write is completed. Usually there is no delay, but if there is it will only be as long as required for database activity to slow to roughly half capacity. The user sees no wait, which improves the feel of your site.

If the server is taken down without the write being completed, it will be completed the next time the server is brought up. Cached Write operations are stored on disk until completed.

Cached Write operations are exactly like any other write operation, except that the operation is signalled by W:CI (insert only).

©Copyright 2023, ZeroPoint, LLC.      Questions? Comments? Email Us.