If your database allows expressions in the index, you can do something like this (ANSI SQL):
CREATE UNIQUE INDEX on your_table (least(column1, column2)
, greatest(column1, column2));
Please note that this is a unique index , not a unique restriction. The only difference for most DBMSs is that you cannot have a unique index as the target of the foreign key, but otherwise they fulfill the same goal.
If your DBMS does not have least()or greatest(), you can replace it with the CASE expression:
create unique index on your_table
(case column1 < column2 then column1 else column2 end,
case column2 > column1 then column2 else column1 end));