This page is divided into two parts. The first parts describes the opcodes I have found out myself due to the lack of any information.
Thanks to the IBM museum in Sindelfingen (near Stuttgart), Germany, I was able to find the official description of the machine language. This is listed in the second part.
Instructions or functions that I couldn't find out or that I have guessed wrong have been marked with a (**) and corrected or added. I personally prefer my own mnemonics because they are much simpler and easier to write and memorize.

In addition to this I have found a detailed description (also coming from the IBM museum) of each instruction in an IBM paper.
There was only chapter 2, so if someone recognizes the text and should have the remaining parts please contact me.

Opcodes part 1

All machine instructions and mnemonics as well as their meanings have been reverse engineered by hand because no information has been available.

List of the maschine instructions for the IBM 5100, 5110 and 5120

Hex    Binary                  Mnemonic           Description                                  
0xy0   0000xxxxyyyy0000   DEC2 Rx, Ry        Rx <- Ry - 2
0000   0000000000000000   HALT               Endless loop with DEC2 R0, R0
0xy1   0000xxxxyyyy0001   DEC Rx, Ry         Rx <- Ry - 1
0xy2   0000xxxxyyyy0010   INC Rx, Ry         Rx <- Ry + 1
0xy3   0000xxxxyyyy0011   INC2 Rx, Ry        Rx <- Ry + 2
0xy4   0000xxxxyyyy0100   MOVE Rx, Ry        Rx <- Ry
0004   0000000000000100   NOP                does a MOVE R0, R0
0xy5   0000xxxxyyyy0101   AND Rx, Ry         Rx <- Lo(Rx) AND Lo(Ry)
0xy6   0000xxxxyyyy0110   OR Rx, Ry          Rx <- Lo(Rx) OR Lo(Ry)
0xy7   0000xxxxyyyy0111   XOR Rx, Ry         Rx <- Lo(Rx) XOR Lo(Ry)
0xy8   0000xxxxyyyy1000   ADD Rx, Ry         Rx <- Rx + Lo(Ry)
0xy9   0000xxxxyyyy1001   SUB Rx, Ry         Rx <- Rx - Lo(Ry)
0xyA   0000xxxxyyyy1010   ADDH Rx, Ry        Lo(Rx) <- Hi(Rx) + Lo(Ry)
                                             Hi(Rx) <- Carry ($00 or $01) (**)
0xyB   0000xxxxyyyy1011   ADDH2 Rx, Ry       (**)
                                             Lo(Rx) <- Hi(Rx) + Lo(Ry)
                                             Hi(Rx) <- Carry - 1 ($FF or $00)
0xyC   0000xxxxyyyy1100   MHL Rx, Ry         Lo(Rx) <- Hi(Ry)
0xyD   0000xxxxyyyy1101   MLH Rx, Ry         Hi(Rx) <- Lo(Ry)
0iyE   0000iiiiyyyy1110   GETB Ry, i         Ry <- Data byte from device i
0iyF   0000iiiiyyyy1111   GETADD Ry, i       Ry <- Ry + {0,2,4,6,8,A,C,E}
                                             depending on data byte from device i
                                             (kind of priority decoding) (**)

1ijj   0001iiiijjjjjjjj   CTRL i, #j         Sends control byte j to device i

2xii   0010xxxxiiiiiiii   MOVE Rx, i'        Rx <- (i')        i' = 2*i
3xii   0011xxxxiiiiiiii   MOVE i', Rx        (i') <- Rx        i' = 2*i

4ix0   0100iiiixxxx0000   PUTB i, (Rx)+      Sends data byte addressed via Rx
                                             to device i;
                                                  Rx <- Rx + 1
4ix1   0100iiiixxxx0001   PUTB i, (Rx)++     dto; Rx <- Rx + 2
4ix2   0100iiiixxxx0010   PUTB i, (Rx)+++    dto; Rx <- Rx + 3
4ix3   0100iiiixxxx0011   PUTB i, (Rx)++++   dto; Rx <- Rx + 4
4ix4   0100iiiixxxx0100   PUTB i, (Rx)-      dto; Rx <- Rx - 1
4ix5   0100iiiixxxx0101   PUTB i, (Rx)--     dto; Rx <- Rx - 2
4ix6   0100iiiixxxx0110   PUTB i, (Rx)---    dto; Rx <- Rx - 3
4ix7   0100iiiixxxx0111   PUTB i, (Rx)----   dto; Rx <- Rx - 4
4ix8   0100iiiixxxx1000   PUTB i, (Rx)       dto; Rx remains unaltered

5xy0   0101xxxxyyyy0000   MOVE (Ry)', Rx     (Ry) <- Rx; Ry <- Ry + 1
5xy1   0101xxxxyyyy0001   MOVE (Ry)+, Rx     (Ry) <- Rx; Ry <- Ry + 2
                          MOVE (Ry)+', Rx
                          MOVE (Ry)++, Rx
                          MOVE (Ry)~, Rx     (Ry) <- Rx; Ry <- Ry - 1
                          MOVE (Ry)-, Rx
                          MOVE (Ry)-~, Rx
                          MOVE (Ry)--, Rx
5xy8   0101xxxxyyyy1000   MOVE (Ry), Rx      (Ry) <- Rx

6xy0   0110xxxxyyyy0000   MOVB Rx, (Ry)+     Rx <- Lo((Ry)); Ry <- Ry + 1
...    ...                ...
6xy8   0110xxxxyyyy1000   MOVB Rx, (Ry)      Rx <- Lo((Ry))

7xy0   0111xxxxyyyy0000   MOVB (Ry)+, Rx     (Ry) <- Lo(Rx); Ry <- Ry + 1
...    ...                ...
7xy8   0111xxxxyyyy1000   MOVB (Ry), Rx      (Ry) <- Lo(Rx)

8xii   1000xxxxiiiiiiii   LBI Rx, #i         Rx <- i

9xii   1001xxxxiiiiiiii   CLR Rx, #i         Rx <- Rx AND NOT i;
                                             Clears all bits in Rx that are
					     set in i

Axii   1010xxxxiiiiiiii   ADD Rx, #(i+1)     Rx <- Rx + (i + 1)

Bxii   1011xxxxiiiiiiii   SET Rx, #i         Rx <- Rx OR i;
                                             Sets all bits in Rx that are
					     set in i

Cxy0   1100xxxxyyyy0000   SLE Rx, Ry         Skip if Lo(Rx) <= Lo(Ry)
Cxy1   1100xxxxyyyy0001   SLT Rx, Ry         Skip if Lo(Rx) < Lo(Ry)
Cxy2   1100xxxxyyyy0010   SE Rx, Ry          Skip if Lo(Rx) = Lo(Ry)
Cx03   1100xxxx00000011   SZ Rx              Skip if Lo(Rx) = 0
Cx04   1100xxxx00000100   SS Rx              Skip if Lo(Rx) = $FF
Cxy5   1100xxxxyyyy0101   SBS Rx, Ry         Skip if all set bits in Ry are
                                             also set in Rx                
Cxy6   1100xxxxyyyy0110   SBC Rx, Ry         Skip if all set bits in Ry
                                             are cleared in Rx
Cxy7   1100xxxxyyyy0111   SBSH Rx, Ry        Skip if all set bits in Ry are
                                             also set in Hi(Rx)
Cxy8   1100xxxxyyyy1000   SGT Rx, Ry         Skip if Lo(Rx) > Lo(Ry)
Cxy9   1100xxxxyyyy1001   SGE Rx, Ry         Skip if Lo(Rx) >= Lo(Ry)
CxyA   1100xxxxyyyy1010   SNE Rx, Ry         Skip if Lo(Rx) <> Lo(Ry)
Cx0B   1100xxxx00001011   SNZ Rx             Skip if Lo(Rx) <> 0
Cx0C   1100xxxx00001100   SNS Rx             Skip if Lo(Rx) <> $FF
CxyD   1100xxxxyyyy1101   SNBS Rx, Ry        Skip if not all set bits in Ry
                                             are set in Rx
, too
CxyE   1100xxxxyyyy1110   SNBC Rx, Ry        Skip if not all set bits in Ry
                                             are cleared in Rx
CxyF   1100xxxxyyyy1111   SNBSH Rx, Ry       Skip if not all set bits in Ry
                                             are set in Hi(Rx), too

Dx01   1101xxxx00000001   LWI Rx, #i         Rx <- i, the same as
iiii   iiiiiiiiiiiiiiii
                      MOVE Rx, (R0)+ ; dw i

Dxy0   1101xxxxyyyy0000   MOVE Rx, (Ry)'     Rx <- (Ry); Ry <- Ry + 1
...    ...                ...
Dxy8   1101xxxxyyyy1000   MOVE Rx, (Ry)      Rx <- (Ry)

E0xC   11100000xxxx1100   SHR Rx             Shift Lo(Rx) one bit to the right
                                             (**) The LSB of Hi(Rx) is taken as
					          new MSB of Lo(Rx)
					          (arithmetic shift)
E0xD   11100000xxxx1101   ROR Rx             Rotate Lo(Rx) one bit to the right
E0xE   11100000xxxx1110   ROR3 Rx            (**) Rotate Lo(Rx) 3 bits to the right
E0xF   11100000xxxx1111   SWAP Rx            Swap the two nibbles of Lo(Rx)
                                             (**) Rotate Lo(Rx) 4 bits

Eix0   1110iiiixxxx0000   GETB (Rx)+, i      Read a data byte from device i
                                             to (Rx); Rx <- Rx + 1
...    ...                ...
Eix8   1110iiiixxxx1000   GETB (Rx), i       dto.
; Rx remains unchanged

EixF   1110iiiixxxx1111   STAT Rx, i         Read status byte from device i
                                             into Rx

Fxii   1111xxxxiiiiiiii   SUB Rx, #(i+1)     Rx <- Rx - (i + 1)

Opcodes part 2

The machine instructions are always 16-bit words (IBM calls this halfword). The first four bits indicate the opcode, the remaining 12 bits depend on the opcode.


Opcode    2. Hex no.     3. Hex no.     4. Hex no.      Instructions                       
  0            Rx            Ry            AM           ADD, ADDS1, ADDS2, AND, GETA, GETR,
                                                        HTL, LTH, MOVE, MVM1, MVM2, MVP1,
                                                        ORB, SUB, XOR
  1            DA                  Command              CTL
  2            Rx                  Address              LDHD
  3            Rx                  Address              STHD
  4            DA            Ry            M            PUTB
  5            Rx            Ry            M            STHI
  6            Rx            Ry            M            LDBI
  7            Rx            Ry            M            STBI
  8            Rx                  Value                EMIT
  9            Rx                  Bit mask             CLRI
  A            Rx                  Value                ADDI
  B            Rx                  Bit mask             SETI
  C            Rx            Ry            JM           All jump instructions
  D            Rx            Ry            M            LDHI
  E            DA            Ry            SM           ROTR, SHFTR, GETB, GETRB
  F            Rx                  Value                SUBI


Name   Value   Description                                                             
Normal modifier
 M     0 - 3   1 - 4 is added to the contents of Ry
       4 - 7   1 - 4 is subtracted from the contents of Ry
       > 7     Ry remains unchanged 

ALU modifier
 AM    0 - F   selects one of 16 ALU functions

Jump modifier
 JM    0 - F   selects one of 16 conditional jumps

Special modifier
 SM    0 - B   only GETB instruction: like modifier M
       C - F   selects GETRB instruction (called STAT in part 1)
               Device address 0 offers the rotate and shift instructions.

Alphabetical list

Mnemonic       Name                            Opcode   Category                
ADD            Add                              0xy8    Arithmetical register
ADDI           Add immediate                    Axii    Arithmetical register
ADDS1          Add special 1                    0xyA    Arithmetical register
ADDS2          Add special 2                    0xyB    Arithmetical register
AND            And                              0xy5    Logical register
CLRI           Clear immediate                  9xii    Logical register
CTL            Control                          1ijj    I/O
EMIT           Emit byte                        8xii    Logical register
GETA           Get add                          0xyF    I/O
GETB           Get byte                         Eix?    I/O
GETR           Get to register                  0iyE    I/O
GETRB          Get byte to register             EixF    I/O
HTL            High to low                      0xyC    Register
JALL           Jump all ones                    Cx04    Jump
JALLM          Jump all masked                  Cxy5    Jump
JEQ            Jump equal                       Cxy2    Jump
JHAM           Jump high all masked             Cxy7    Jump
JHE            Jump high or equal               Cxy9    Jump
JHI            Jump high                        Cxy8    Jump
JHL            Jump high or low (not equal)     CxyA    Jump
JHSNM          Jump high some bit not masked    CxyF    Jump
JLE            Jump low or equal                Cxy0    Jump
JLO            Jump low                         Cxy1    Jump
JNO            Jump no ones                     Cx03    Jump
JNOM           Jump no ones masked              Cxy6    Jump
JSB            Jump some bits                   Cx0B    Jump
JSM            Jump some masked                 CxyE    Jump
JSN            Jump some not ones               Cx0C    Jump
JSNM           Jump some not masked             CxyD    Jump
LDBI           Load byte indirect               6xy?    Fetch&Store
LDHD           Load halfword direct             2xii    Fetch&Store
LDHI           Load halfword indirect           Dxy?    Fetch&Store
LTH            Low to high                      0xyD    Register
MOVE           Move register                    0xy4    Register
MVM1           Move minus 1                     0xy1    Register
MVM2           Move minus 2                     0xy0    Register
MVP1           Move plus 1                      0xy2    Register
MVP2           Move plus 2                      0xy3    Register
ORB            Or byte                          0xy6    Logical register
PUTB           Put byte                         4ix?    I/O
ROTR           Rotate register                  E0x?    Logical register
SETI           Set immediate                    Bxii    Logical register
SHFTR          Shift right                      E0xC    Logical register
STBI           Store byte indirect              7xy?    Fetch&Store
STHD           Store halfword direct            3xii    Fetch&Store
STHI           Store halfword indirect          5xy?    Fetch&Store
SUB            Subtract                         0xy9    Arithmetical register
SUBI           Subtract immediate               Fxii    Arithmetical register
XOR            Exclusive or                     0xy7    Logical register