SAP ABAP Cluster problem: the (modern) solution

Hi Folks,

well the main problem with data clusters is that you store a binary representation of the content of your data structures and ddic data at the TIME OF SAVING THE CLUSTER
If something changes later on you are on your own.

A good approach to save data in clusters is to save data in the cluster in another form that can be easily written and read from your programs and put in the abap data structures.

In modern times this could well be called "serialization" and "deserialization" of data.

It works like this.
You take your tables and structures.
Transform them into an XML string.
The xml string can be a binary string in abap so if ddic and the like changes (up to a certain degree of change) you get no effects.
You write the xml to the cluster.

Now if you change the data structure when deserializing the xml most unwanted effects with the binary representations of data in clusters disappear.

For example adding a new field in between is harmless: the transformation will leave it empty instead of dumping.
Of course if you change the data type of a char to numeric or worse floating point you are likely to get trash out of your data or a dump. But this is normal in some way.

Here is a snippet of coding you can use as an example

*old structure
TYPES: BEGIN OF struc1,
         f1 TYPE c LENGTH 2,
         f2 TYPE c LENGTH 2,
         f3 TYPE c LENGTH 2,
       END OF struc1.

*new structure
TYPES: BEGIN OF struc2,
         fnew  TYPE c LENGTH 2,
         f1    TYPE c LENGTH 5,
         f2    TYPE i,
       END OF struc2.

*data tables
DATA gt_tab1 TYPE STANDARD TABLE OF struc1.
DATA gt_tab2 TYPE STANDARD TABLE OF struc2.
*id for storing data
DATA gc_id    TYPE c LENGTH 20 VALUE 'MYKEY'.
* XML-String c
DATA gv_xml   TYPE string.

DATA gx_error TYPE REF TO cx_..._check.

START-OF-SELECTION.


  APPEND 'ZZ1122' TO gt_tab1.
  APPEND 'EE2233' TO gt_tab1.
  APPEND 'DD3344' TO gt_tab1.

*transform table into xml
  CALL TRANSFORMATION id
       SOURCE test = gt_tab1
       RESULT XML gv_xml.

*save xml to data cluster
  EXPORT test = gv_xml TO DATABASE indx(zr) ID gc_id.

  CLEAR gt_tab1.
  CLEAR gt_tab2.
  CLEAR gv_xml.

*import data to xml
  IMPORT test TO gv_xml FROM DATABASE indx(zr) ID gc_id.
 
  TRY .
* transform xml data with NEW structure
      CALL TRANSFORMATION id
           SOURCE XML gv_xml
           RESULT test = gt_tab2.
    CATCH cx_transformation_error INTO gx_error.
 *Does not DUMP!!!
  ENDTRY.

SAP ABAP cluster problem: the workaround and the solution

Hi All,
In my previous post i showed how a change in the cluster data structures can bring up big problem.
Here i will try to explore a solution.

Solution 1: the good old time's solution: add the new field to the end.

If you change the structure like this:


TYPES: BEGIN OF struc,
         procedure TYPE c LENGTH 10,
         sernr   TYPE c LENGTH 13,
         prat TYPE c LENGTH 12,
  END OF struc.

It works just fine.

So as long as you add a new field at the very very end of everithing you write to the cluster you can do it.

Of course if you have a cluster holding several tables and you want to add a field to one of the tables not the last. You cannot do it. You need to do a conversion. Here we go.

Solution 2:
run a conversion report.
You write a report like this on.

TYPES: BEGIN OF struc_new,
         procedure TYPE c LENGTH 10,
         prat TYPE c LENGTH 12,
         sernr   TYPE c LENGTH 13,
END OF struc_new.

TYPES: BEGIN OF struc,
         procedure TYPE c LENGTH 10,         prat TYPE c LENGTH 12,
         sernr   TYPE c LENGTH 13,

  END OF struc.

DATA gt_mydata type standard table of struc.
DATA gt_mydatanew type standard table of struc_new.

data ls_struc type struc.
data ls_struc_new type struc_new.

import gt_mydata from DATABASE indx(ZR) ID 'MYKEY'.

 loop at gt_mydata into ls_struc.
  move corresponding ls_struc to ls_struc_new.
  append ls_struc_new to gt_mydata_new.
endloop.

export gt_mydata_new to DATABASE indx(ZR) ID 'MYKEY'.

From now on the import export will work with the new structure, but not with the old structure.

This is basically what the good old xpra programs did.

There is also a better solution which i would call "the modern solution".
More in the next post about this.

SAP ABAP Cluster table problem example

Hi Folks,
my previous post  might sound a bit cryptic.
I want to make an example of the havoc that can happen when you use cluster tables.

Suppose you have decided to store in a cluster table the following data

TYPES: BEGIN OF struc,
         procedure TYPE c LENGTH 10,
         sernr   TYPE c LENGTH 13,
       END OF struc.

 You do 

EXPORT gt_mydata TO DATABASE indx(ZR) ID 'MYKEY'.

Then a couple of years pass and your cluster indx RELID ZR is full of your stored procedure numbers with material numerbs and serial numbers.
But because of a change in company procedures you need to store from today on a new number the "processing number".

What any programmer would do is to extend the structure struc like this


TYPES: BEGIN OF struc,
         procedure TYPE c LENGTH 10,
         procnr  type c linght 10
         sernr   TYPE c LENGTH 13,
       END OF struc.

If now you try to import your previously exported data you get a DUMP.
Yes a dump.
IMPORT gt_mydata from DATABASE INDX(ZR) ID 'MYKEY'.
will return CX_SY_IMPORT_MISMATCH_ERROR 

Now this example is trivial. But imagine you use a ddic type and someone (maybe SAP or an external provider) changes tha ddic type not knowing that it is used in a cluster. 
What happens?
How can one solve this problem?

The answer in my next post.

  

The truth about SAP ABAP cluster tables

Hi folks,
Todays argument is cluster tables.
In the internet you find mainly a lot of confusion, starting from the sap help portal which states:
"Table Pool: A table in the database in which all records from the pooled tables assigned to the table pool are stored corresponds to a table pool."
And "Cluster TABLE: Several logical data records from different cluster tables can be stored together in one physical record in a table cluster.

LOL: It might be my poor english but imho it is more or less like saying a table cluster is a cluster of tables and a table pool is a pool of tables.

I'll try to stick with the cluster tables.

To keep it simple assume you are a bookkeeper for a veterinarian  ( the doctor who cares for the health of animals ) suppose working with dogs only and you have to keep track of his patients data .

You can have several booklets (tables) where you write different things:

• One book for the money paid by each person for its dog (key dog name, chip-id)
• One book where you write name and adrress of the person the dog belongs to (key dog name, chip-id)
• One book where you write the illnesses (key dog name, chip-id)
• One book for the medicines (key dog name, chip-id)
• One book to keep track the individual visits (key dog name, chip-id)

An alternative could be to have a single book where you enter for the key (dog name,chip-id) all the data you need, placing sequentially after the name / chip id tag the different data related to the single dog.

In the first case you have different places ( transparent tables ) where you put data with the same key.
In the second case (cluster table) you have a single table where under a single key you save several different data stored. Even tables structures etc.

Technically speacking, from a programming point of view, and logically it is quite a big difference.

One might say that it is quite confortable to have all data available for a given key without selecting from many tables.
On the other hand, once one knows that the cluster table cannot be (SQL) selected on other then using the main key ( dog-chip id ) because it is compressed and saved in a row compressed (binary) format, one realizes that informations like "dogs sick this month" "medicines prescribed this month" etc. are not really accessible at a glance.
Maybe this is the reason why normal tables are called "transparent" (i guess)... the cluster tables could well be called "obscured tables" or "dark tables" because you do not really know what is inside, and if you do not really know what is inside ( i mean the abap structure definitions for the import export statements the exact sequence of the fields and tables in the export) you have really few chances of extracting this data.

someone uses this as an argument pro-cluster: the data is compressed and not readable at the database level. This means security for them.
Someone else who faced an upgrade where a ddic data structure changed faced the dark side of the security of cluster tables: corrupt data that cannot be retrieved. Let alone the case where someone switched unicode on on a system... i dont' dare thinking of what happens. this is because the data is saved in a binary form in the cluster. if the size of the ddic structures you read from the cluster changes and the cluster is not converted before you read corrupt data.

it is like writing a 16 bit integer ( to keep it simple ) and later reading 32 bit. You are likely to read 16 correct bits and 16 bits of junk.

To be more precise a cluster table contains a compressed set of binary data which reflect the internal representation of the corresponding abap data structures at the time of writing the cluster.

If the binary structure of this binary representation changes later on for whatever reason ( change in the programming language, endianess of the machine, unicode on/off, different codepages, structure boundaries,  new fields in table of in the export statements etc etc) you get corrupt data.

The impression is that as soon as SAP realizes the problems that arise they go for a solution at least they did in the past. For example now there is a check on the data type stored in the cluster. probably now they implemented conversion mechanism in case something changes. Whenever they change something in the cluster tables contents they release xpra programs to massively update cluster contents.

So there are many possible pitfalls when using cluster. And many attentions to use in practice.
Ill'come back to this topics in later posts.

Now let's try a definition:

Cluster tables are database tables that combine the data from several tables with identical keys into one physical record on the database.

    Data is written to a database in compressed form by means of special commands of the abap programming language.
    Retrieval of data is very fast (your mileage might vary).
    Cluster tables are defined in the data dictionary as transparent tables.
    External programs can NOT interpret the data in a cluster table.
    Special language elements EXPORT TO DATABASE, IMPORT TO DATABASE and DELETE  FROM DATABASE are used to process data in the cluster tables.

In HR there are well know cluster tables:

PCL1 - Database for HR work area;
PCL2 - Accounting Results (time, payroll);
PCL3 - Applicant tracking data
PCL4 - Documents  (change logs, etc)

Here is a typical cluster table ( in the abap dictionary )


Field    Data Element    Data Type    length    Checktable    Description
MANDT    MANDT    CLNT    000003    *    Client
RELID    INDX_RELID    CHAR    000002        Area
SRTFD    INDX_SRTFD    CHAR    000022        ID
SRTF2    INDX_SRTF2    INT4    000010        Counter
LOEKZ    SYCHAR01    CHAR    000001       
SPERR    SYCHAR01    CHAR    000001       
AEDAT    SYDATS    DATS    000008       
USERA    USERNAME    CHAR    000012        User name
PGMID    PROGNAME    CHAR    000040        Program Name
BEGDT    SYDATS    DATS    000008       
ENDDT    SYDATS    DATS    000008       
CLUSTR    INDX_CLSTR    INT2    000005        Data length
CLUSTD    INDX_CLUST    LRAW    002886        Data

Your key is saved typically in the srtfd field
Your data is saved in the "raw field" CLUSTD.
If the data is too much for a single raw field the abap language automatically creates as many lines with the same key (incrementing field srtf2) as needed.
Relid represent is the main key. In one cluster table you can have several different data clusters saved.

For example in table PCL1 you have cluster RELID = TE and cluster RELID = PC. This means that in the same table SAP stores data for travel exenses under the main key "TE" and personal calendar data under the index "PC".

I understand that cluster tables are a complex topic. Instead of trying to read write data from a new relid in the indx table we have tried understanding the real implications of creating a cluster table and the risks involved. SAP has troops of developers to take care of their cluster tables. In a company with custom developments based on a cluster the situation might be different. The developers might not be aware of the pitfalls and changes in systems settings/upgrades/SPs might cause unwanted devastating side effects on your data in the cluster.

Conclusion: I would suggest that unless you are not really sure of what you are doing and you know how to handle its maintainance during the years i would discard the idea of creating a new cluster table and i would go for a transparent database table design.

How to change table data from se16n with "/h" (in debug) in SAP R/3

Hi Folks,
some developer prefers to change table data in se16n instead of se30 or se16 (see my previous post about changing data from se16 in debug).

The procedure is very simple.
- Start se16n, and enter the table (for example t001p)
- Start the debugger with /h
- Type the following variables in the variable table:
gd-sap-edit
gd-edit
-Modify them changing the value to X


-press f8

- voilà you can change everything! :)



Have fun and be careful with what you change!!

ps. the trick is all in the structure gd give it a look you might find some other interesting fields to change. I'm curious what tabedit does!?

How to change table data from se16 with "/h" (in debug) in SAP R/3

Hi Folks,
this is also an easy one. But it is very useful.
To change data from se16 even if the table is set as non changeable in se11 you can do the following.

1) From Menu -> user settings select the "standard se16 list". This procedure does not work with the grid or alv view.

2) Select the record you want to change by marking it individually




3) double click on the selected record. You get this

4) Put /h in the trasaction box and press enter

5) the debugger starts. Look for "EDIT" in the code. double click on the "code" variable.




6) change "code" variable to "EDIT" in the debugger.










6a) type "code" in the variables tables.

6b) Press the pencil button,

6c) type"EDIT" instead of show and press "ENTER"




7) press F8 ( run the program )

8) Voilà the list changed to edit mode 




Limitations of this method
1) This method should not be run in production systems. And not use to tamper data.
2) You need debugging authorization ( changes in debugging are logged in sm21 and many sysops run log scanners on sm21 )
3) Some tables don't work fine because they have external key and the se16 blocks you. ( for example table pa0001).

In addition you can experiment with the other sy.ucomm values that you find in the coding. Like "ANVO" "INSR" etc... they do interesting things like copying a record, inserting a new one and some other.